Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 1454319 Details for
Bug 1594698
[mixed-versions] Create volume fails on compat installation {UUID} failed to build and is in ERROR status
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
volume.log from the controller
volume.log (text/plain), 11.05 MB, created by
Ronnie Rasouli
on 2018-06-25 10:50:42 UTC
(
hide
)
Description:
volume.log from the controller
Filename:
MIME Type:
Creator:
Ronnie Rasouli
Created:
2018-06-25 10:50:42 UTC
Size:
11.05 MB
patch
obsolete
>2018-06-24 13:00:24.220 139116 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports. >2018-06-24 13:00:24.221 139116 DEBUG oslo_concurrency.lockutils [-] Acquired semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:212 >2018-06-24 13:00:24.221 139116 DEBUG oslo_concurrency.lockutils [-] Releasing semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:225 >2018-06-24 13:00:24.611 139116 DEBUG oslo_db.sqlalchemy.engines [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:285 >2018-06-24 13:00:24.766 139116 INFO cinder.rpc [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] Automatically selected cinder-scheduler objects version 1.27 as minimum service version. >2018-06-24 13:00:24.771 139116 INFO cinder.rpc [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] Automatically selected cinder-scheduler RPC version 3.7 as minimum service version. >2018-06-24 13:00:24.778 139116 DEBUG oslo_db.sqlalchemy.engines [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:285 >2018-06-24 13:00:24.786 139116 INFO cinder.volume.manager [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] Service not found for updating active_backend_id, assuming default for driver init. >2018-06-24 13:00:24.787 139116 WARNING cinder.keymgr.conf_key_mgr [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] This key manager is insecure and is not recommended for production deployments: ServiceNotFound: Service cinder-volume could not be found on host hostgroup@tripleo_ceph. >2018-06-24 13:00:24.810 139116 INFO cinder.volume.manager [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] Image-volume cache disabled for host hostgroup@tripleo_ceph. >2018-06-24 13:00:24.816 139116 DEBUG oslo_db.sqlalchemy.engines [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:285 >2018-06-24 13:00:24.829 139116 INFO oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] Starting 1 workers >2018-06-24 13:00:24.834 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] Started child 139328 _start_child /usr/lib/python2.7/site-packages/oslo_service/service.py:513 >2018-06-24 13:00:24.840 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] Full set of CONF: wait /usr/lib/python2.7/site-packages/oslo_service/service.py:580 >2018-06-24 13:00:24.841 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] ******************************************************************************** log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2865 >2018-06-24 13:00:24.841 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] Configuration options gathered from: log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2866 >2018-06-24 13:00:24.841 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] command line args: ['--config-file', '/usr/share/cinder/cinder-dist.conf', '--config-file', '/etc/cinder/cinder.conf', '--logfile', '/var/log/cinder/volume.log'] log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2867 >2018-06-24 13:00:24.842 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] config files: ['/usr/share/cinder/cinder-dist.conf', '/etc/cinder/cinder.conf'] log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2869 >2018-06-24 13:00:24.842 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] ================================================================================ log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2870 >2018-06-24 13:00:24.843 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] allow_availability_zone_fallback = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.842 139328 INFO cinder.service [-] Starting cinder-volume node (version 11.1.0) >2018-06-24 13:00:24.843 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] allowed_direct_url_schemes = [] log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.844 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] api_rate_limit = True log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.844 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] auth_strategy = keystone log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.845 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_availability_zone = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.845 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_host = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.847 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backup_api_class = cinder.backup.api.API log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.848 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backup_manager = cinder.backup.manager.BackupManager log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.849 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backup_name_template = backup-%s log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.849 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backup_use_same_host = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.849 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backup_use_temp_snapshot = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.850 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] chap_password = **** log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.850 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] chap_username = log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.851 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] chiscsi_conf = /etc/chelsio-iscsi/chiscsi.conf log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.852 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] cinder_internal_tenant_project_id = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.852 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] cinder_internal_tenant_user_id = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.852 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] cluster = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.853 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] compute_api_class = cinder.compute.nova.API log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.853 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] config_dir = [] log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.854 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] config_file = ['/usr/share/cinder/cinder-dist.conf', '/etc/cinder/cinder.conf'] log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.854 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] consistencygroup_api_class = cinder.consistencygroup.api.API log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.855 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] control_exchange = openstack log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.855 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] db_driver = cinder.db log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.856 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] debug = True log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.856 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] default_availability_zone = nova log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.857 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] default_group_type = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.858 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'dogpile.core.dogpile=INFO'] log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.858 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] default_volume_type = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.858 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] driver_client_cert = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.859 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] driver_client_cert_key = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.859 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] driver_data_namespace = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.860 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] driver_ssl_cert_path = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.860 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] driver_ssl_cert_verify = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.861 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] driver_use_ssl = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.861 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] enable_new_services = True log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.861 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] enable_unsupported_driver = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.862 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] enable_v1_api = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.862 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] enable_v2_api = True log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.863 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] enable_v3_api = True log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.863 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] enabled_backends = ['tripleo_ceph'] log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.863 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] enforce_multipath_for_image_xfer = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.864 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] fatal_exception_format_errors = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.864 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] filter_function = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.865 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] glance_api_insecure = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.866 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] glance_api_servers = ['http://172.17.1.12:9292'] log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.866 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] glance_api_ssl_compression = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.867 139116 WARNING oslo_config.cfg [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] Option "glance_api_version" from group "DEFAULT" is deprecated for removal (Glance v1 support will be removed in Queens). Its value may be silently ignored in the future. >2018-06-24 13:00:24.868 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] glance_api_version = 2 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.868 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] glance_ca_certificates_file = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.869 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] glance_catalog_info = image:glance:publicURL log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.869 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] glance_core_properties = ['checksum', 'container_format', 'disk_format', 'image_name', 'image_id', 'min_disk', 'min_ram', 'name', 'size'] log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.870 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] glance_num_retries = 0 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.870 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] glance_request_timeout = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.870 139328 INFO cinder.volume.manager [req-5412b44b-7b69-4daa-a560-9093d875a059 - - - - -] Including all resources from host hostgroup@tripleo_ceph in cluster None. >2018-06-24 13:00:24.871 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] goodness_function = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.872 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] graceful_shutdown_timeout = 60 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.872 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] group_api_class = cinder.group.api.API log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.874 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] host = hostgroup log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.875 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] iet_conf = /etc/iet/ietd.conf log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.876 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] image_conversion_dir = /var/lib/cinder/conversion log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.877 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] image_upload_use_cinder_backend = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.877 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] image_upload_use_internal_tenant = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.878 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] image_volume_cache_enabled = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.878 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] image_volume_cache_max_count = 0 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.879 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] image_volume_cache_max_size_gb = 0 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.879 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] instance_format = [instance: %(uuid)s] log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.880 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] instance_uuid_format = [instance: %(uuid)s] log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.880 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] iscsi_helper = lioadm log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.881 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] iscsi_iotype = fileio log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.882 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] iscsi_ip_address = 10.0.0.101 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.882 139328 DEBUG oslo_db.sqlalchemy.engines [req-5412b44b-7b69-4daa-a560-9093d875a059 - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:285 >2018-06-24 13:00:24.882 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] iscsi_port = 3260 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.883 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] iscsi_protocol = iscsi log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.884 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] iscsi_secondary_ip_addresses = [] log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.884 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] iscsi_target_flags = log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.885 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] iscsi_target_prefix = iqn.2010-10.org.openstack: log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.885 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] iscsi_write_cache = on log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.886 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] iser_helper = tgtadm log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.887 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] iser_ip_address = 10.0.0.101 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.887 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] iser_port = 3260 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.888 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] iser_target_prefix = iqn.2010-10.org.openstack: log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.888 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] log_config_append = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.889 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] log_date_format = %Y-%m-%d %H:%M:%S log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.890 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] log_dir = /var/log/cinder log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.891 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] log_file = /var/log/cinder/volume.log log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.891 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] log_options = True log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.892 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] logging_context_format_string = %(asctime)s.%(msecs)03d %(process)d %(levelname)s %(name)s [%(request_id)s %(user_identity)s] %(instance)s%(message)s log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.892 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] logging_debug_format_suffix = %(funcName)s %(pathname)s:%(lineno)d log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.893 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] logging_default_format_string = %(asctime)s.%(msecs)03d %(process)d %(levelname)s %(name)s [-] %(instance)s%(message)s log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.893 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] logging_exception_prefix = %(asctime)s.%(msecs)03d %(process)d ERROR %(name)s %(instance)s log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.894 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] logging_user_identity_format = %(user)s %(tenant)s %(domain)s %(user_domain)s %(project_domain)s log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.894 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] max_age = 0 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.895 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] max_over_subscription_ratio = 20.0 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.895 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] message_reap_interval = 86400 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.896 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] message_ttl = 2592000 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.896 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] migration_create_volume_timeout_secs = 300 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.897 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] monkey_patch = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.897 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] monkey_patch_modules = [] log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.898 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] my_ip = 10.0.0.101 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.898 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] no_snapshot_gb_quota = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.898 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] num_iser_scan_tries = 3 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.899 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] num_shell_tries = 3 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.899 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] num_volume_device_scan_tries = 3 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.899 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] os_privileged_user_auth_url = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.900 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] os_privileged_user_name = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.900 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] os_privileged_user_password = **** log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.901 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] os_privileged_user_tenant = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.901 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] osapi_max_limit = 1000 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.902 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] osapi_volume_base_URL = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.902 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] osapi_volume_ext_list = [] log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.902 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] osapi_volume_extension = ['cinder.api.contrib.standard_extensions'] log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.903 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] osapi_volume_listen = controller-0.internalapi.localdomain log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.903 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] osapi_volume_listen_port = 8776 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.903 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] osapi_volume_use_ssl = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.904 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] osapi_volume_workers = 4 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.904 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] per_volume_size_limit = -1 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.905 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] periodic_fuzzy_delay = 60 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.905 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] periodic_interval = 60 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.905 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] public_endpoint = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.905 139328 INFO cinder.volume.manager [req-5412b44b-7b69-4daa-a560-9093d875a059 - - - - -] 0 volumes, 0 consistency groups, 0 generic groups and 0 image volume caches from host hostgroup@tripleo_ceph have been included in cluster None. >2018-06-24 13:00:24.906 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] publish_errors = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.906 139328 INFO cinder.volume.manager [req-5412b44b-7b69-4daa-a560-9093d875a059 - - - - -] Starting volume driver RBDDriver (1.2.0) >2018-06-24 13:00:24.906 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] query_volume_filters = ['name', 'status', 'metadata', 'availability_zone', 'bootable', 'group_id'] log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.906 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] quota_backup_gigabytes = 1000 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.907 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] quota_backups = 10 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.907 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] quota_consistencygroups = 10 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.907 139328 DEBUG cinder.volume.drivers.rbd [req-5412b44b-7b69-4daa-a560-9093d875a059 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:00:24.908 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] quota_driver = cinder.quota.DbQuotaDriver log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.908 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] quota_gigabytes = 1000 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.908 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] quota_groups = 10 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.909 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] quota_snapshots = 10 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.909 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] quota_volumes = 10 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.909 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] rate_limit_burst = 0 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.910 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] rate_limit_except_level = CRITICAL log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.910 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] rate_limit_interval = 0 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.911 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] replication_device = **** log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.911 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] report_discard_supported = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.912 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] report_interval = 10 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.912 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] reservation_clean_interval = 86400 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.912 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] reservation_expire = 86400 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.913 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] reserved_percentage = 0 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.913 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] resource_query_filters_file = /etc/cinder/resource_filters.json log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.913 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] rootwrap_config = /etc/cinder/rootwrap.conf log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.914 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] rpc_backend = rabbit log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.914 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] rpc_response_timeout = 60 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.915 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] run_external_periodic_tasks = True log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.915 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] scheduler_manager = cinder.scheduler.manager.SchedulerManager log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.915 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] scst_target_driver = iscsi log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.916 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] scst_target_iqn_name = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.916 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] service_down_time = 60 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.916 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] snapshot_name_template = snapshot-%s log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.917 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] state_path = /var/lib/cinder log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.917 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] storage_availability_zone = nova log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.917 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] storage_protocol = iscsi log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.918 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] syslog_log_facility = LOG_USER log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.918 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] trace_flags = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.919 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] transfer_api_class = cinder.transfer.api.API log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.919 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] transport_url = **** log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.919 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] until_refresh = 0 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.920 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] use_chap_auth = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.920 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] use_default_quota_class = True log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.921 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] use_journal = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.921 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] use_multipath_for_image_xfer = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.921 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] use_stderr = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.922 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] use_syslog = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.922 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] volume_api_class = cinder.volume.api.API log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.923 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] volume_backend_name = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.923 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] volume_clear = zero log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.924 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] volume_clear_ionice = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.924 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] volume_clear_size = 0 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.925 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] volume_copy_blkio_cgroup_name = cinder-volume-copy log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.925 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] volume_copy_bps_limit = 0 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.926 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] volume_dd_blocksize = 1M log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.926 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] volume_manager = cinder.volume.manager.VolumeManager log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.926 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] volume_name_template = volume-%s log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.927 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] volume_service_inithost_offload = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.927 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] volume_usage_audit_period = month log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.928 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] volumes_dir = /etc/cinder/volumes log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.928 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] watch_log_file = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.928 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] zoning_mode = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 >2018-06-24 13:00:24.929 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_messaging_rabbit.amqp_auto_delete = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.930 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_messaging_rabbit.amqp_durable_queues = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.930 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_messaging_rabbit.conn_pool_min_size = 2 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.930 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_messaging_rabbit.conn_pool_ttl = 1200 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.931 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_messaging_rabbit.fake_rabbit = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.931 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_messaging_rabbit.heartbeat_rate = 2 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.932 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_messaging_rabbit.heartbeat_timeout_threshold = 60 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.932 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_messaging_rabbit.kombu_compression = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.933 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_messaging_rabbit.kombu_failover_strategy = round-robin log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.933 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_messaging_rabbit.kombu_missing_consumer_retry_timeout = 60 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.934 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_messaging_rabbit.kombu_reconnect_delay = 1.0 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.935 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_messaging_rabbit.rabbit_ha_queues = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.935 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_messaging_rabbit.rabbit_host = localhost log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.936 139116 WARNING oslo_config.cfg [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] Option "rabbit_port" from group "oslo_messaging_rabbit" is deprecated for removal (Replaced by [DEFAULT]/transport_url). Its value may be silently ignored in the future. >2018-06-24 13:00:24.936 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_messaging_rabbit.rabbit_hosts = ['localhost:5672'] log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.937 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_messaging_rabbit.rabbit_interval_max = 30 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.937 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_messaging_rabbit.rabbit_login_method = AMQPLAIN log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.938 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_messaging_rabbit.rabbit_max_retries = 0 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.938 139116 WARNING oslo_config.cfg [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] Option "rabbit_password" from group "oslo_messaging_rabbit" is deprecated for removal (Replaced by [DEFAULT]/transport_url). Its value may be silently ignored in the future. >2018-06-24 13:00:24.938 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_messaging_rabbit.rabbit_password = **** log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.939 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_messaging_rabbit.rabbit_port = 5672 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.939 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_messaging_rabbit.rabbit_qos_prefetch_count = 0 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.939 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_messaging_rabbit.rabbit_retry_backoff = 2 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.940 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_messaging_rabbit.rabbit_retry_interval = 1 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.940 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_messaging_rabbit.rabbit_transient_queues_ttl = 1800 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.941 139116 WARNING oslo_config.cfg [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] Option "rabbit_userid" from group "oslo_messaging_rabbit" is deprecated for removal (Replaced by [DEFAULT]/transport_url). Its value may be silently ignored in the future. >2018-06-24 13:00:24.941 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_messaging_rabbit.rabbit_userid = guest log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.941 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_messaging_rabbit.rabbit_virtual_host = / log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.942 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_messaging_rabbit.rpc_conn_pool_size = 30 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.943 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_messaging_rabbit.ssl = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.943 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_messaging_rabbit.ssl_ca_file = log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.944 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_messaging_rabbit.ssl_cert_file = log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.945 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_messaging_rabbit.ssl_key_file = log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.945 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_messaging_rabbit.ssl_version = log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.946 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] coordination.backend_url = file:///var/lib/cinder log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.947 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] coordination.heartbeat = 1.0 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.947 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] coordination.initial_reconnect_backoff = 0.1 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.948 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] coordination.max_reconnect_backoff = 60.0 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.949 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] database.backend = sqlalchemy log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.949 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] database.connection = **** log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.950 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] database.connection_debug = 0 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.950 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] database.connection_trace = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.951 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] database.db_inc_retry_interval = True log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.951 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] database.db_max_retries = -1 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.951 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] database.db_max_retry_interval = 10 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.952 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] database.db_retry_interval = 1 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.953 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] database.idle_timeout = 3600 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.953 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] database.max_overflow = 50 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.954 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] database.max_pool_size = 5 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.954 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] database.max_retries = -1 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.955 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] database.min_pool_size = 1 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.955 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] database.mysql_enable_ndb = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.955 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] database.mysql_sql_mode = TRADITIONAL log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.956 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] database.pool_timeout = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.956 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] database.retry_interval = 10 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.957 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] database.slave_connection = **** log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.957 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] database.sqlite_synchronous = True log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.958 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] database.use_db_reconnect = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.959 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_messaging_notifications.driver = ['messagingv2'] log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.959 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_messaging_notifications.retry = -1 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.959 139328 DEBUG cinder.volume.drivers.rbd [req-5412b44b-7b69-4daa-a560-9093d875a059 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:00:24.960 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_messaging_notifications.topics = ['notifications'] log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.960 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_messaging_notifications.transport_url = **** log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.961 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_reports.file_event_handler = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.961 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_reports.file_event_handler_interval = 1 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.961 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_reports.log_dir = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.962 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.backend_availability_zone = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.962 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.backend_host = hostgroup log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.963 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.backup_use_temp_snapshot = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.963 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.chap_password = **** log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.963 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.chap_username = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.964 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.chiscsi_conf = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.964 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.driver_client_cert = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.965 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.driver_client_cert_key = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.965 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.driver_data_namespace = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.965 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.driver_ssl_cert_path = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.966 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.driver_ssl_cert_verify = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.966 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.driver_use_ssl = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.966 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.enable_unsupported_driver = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.967 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.enforce_multipath_for_image_xfer = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.967 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.extra_capabilities = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.967 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.filter_function = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.968 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.goodness_function = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.968 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.iet_conf = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.969 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.image_upload_use_cinder_backend = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.969 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.image_upload_use_internal_tenant = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.969 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.image_volume_cache_enabled = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.970 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.image_volume_cache_max_count = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.970 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.image_volume_cache_max_size_gb = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.971 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.iscsi_helper = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.971 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.iscsi_iotype = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.971 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.iscsi_ip_address = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.972 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.iscsi_port = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.972 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.iscsi_protocol = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.973 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.iscsi_secondary_ip_addresses = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.973 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.iscsi_target_flags = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.974 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.iscsi_target_prefix = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.974 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.iscsi_write_cache = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.975 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.iser_helper = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.975 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.iser_ip_address = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.975 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.iser_port = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.975 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.iser_target_prefix = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.976 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.max_over_subscription_ratio = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.976 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.num_iser_scan_tries = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.976 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.num_shell_tries = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.977 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.num_volume_device_scan_tries = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.977 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.rados_connect_timeout = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.978 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.rados_connection_interval = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.978 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.rados_connection_retries = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.978 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.rbd_ceph_conf = /etc/ceph/ceph.conf log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.979 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.rbd_cluster_name = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.979 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.rbd_flatten_volume_from_snapshot = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.980 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.rbd_keyring_conf = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.980 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.rbd_max_clone_depth = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.981 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.rbd_pool = volumes log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.981 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.rbd_secret_uuid = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.981 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.rbd_store_chunk_size = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.982 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.rbd_user = openstack log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.982 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.replication_connect_timeout = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.982 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.replication_device = **** log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.983 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.report_discard_supported = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.983 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.report_dynamic_total_capacity = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.984 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.reserved_percentage = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.984 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.scst_target_driver = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.984 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.scst_target_iqn_name = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.985 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.storage_protocol = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.985 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.suppress_requests_ssl_warnings = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.986 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.trace_flags = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.987 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.use_chap_auth = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.987 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.use_multipath_for_image_xfer = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.988 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.volume_backend_name = tripleo_ceph log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.988 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.volume_clear = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.989 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.volume_clear_ionice = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.990 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.volume_clear_size = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.990 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.volume_copy_blkio_cgroup_name = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.991 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.volume_copy_bps_limit = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.991 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.volume_dd_blocksize = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.992 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.volume_driver = cinder.volume.drivers.rbd.RBDDriver log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.992 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] tripleo_ceph.volumes_dir = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.993 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] profiler.connection_string = messaging:// log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.993 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] profiler.enabled = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.994 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] profiler.es_doc_type = notification log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.994 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] profiler.es_scroll_size = 10000 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.994 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] profiler.es_scroll_time = 2m log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.995 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] profiler.hmac_keys = SECRET_KEY log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.996 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] profiler.sentinel_service_name = mymaster log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.996 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] profiler.socket_timeout = 0.1 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.997 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] profiler.trace_sqlalchemy = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.997 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] privsep_osbrick.capabilities = [21] log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.998 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] privsep_osbrick.group = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.998 139328 DEBUG cinder.volume.drivers.rbd [req-5412b44b-7b69-4daa-a560-9093d875a059 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:00:24.999 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] privsep_osbrick.helper_command = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:24.999 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] privsep_osbrick.user = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.000 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.backend_availability_zone = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.000 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.backup_use_temp_snapshot = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.001 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.chap_password = **** log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.002 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.chap_username = log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.002 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.chiscsi_conf = /etc/chelsio-iscsi/chiscsi.conf log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.003 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.driver_client_cert = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.003 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.driver_client_cert_key = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.003 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.driver_data_namespace = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.004 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.driver_ssl_cert_path = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.004 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.driver_ssl_cert_verify = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.004 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.driver_use_ssl = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.005 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.enable_unsupported_driver = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.005 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.enforce_multipath_for_image_xfer = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.005 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.extra_capabilities = {} log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.006 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.filter_function = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.006 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.goodness_function = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.006 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.iet_conf = /etc/iet/ietd.conf log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.007 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.image_upload_use_cinder_backend = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.007 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.image_upload_use_internal_tenant = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.007 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.image_volume_cache_enabled = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.008 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.image_volume_cache_max_count = 0 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.008 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.image_volume_cache_max_size_gb = 0 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.008 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.iscsi_helper = tgtadm log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.009 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.iscsi_iotype = fileio log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.009 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.iscsi_ip_address = 10.0.0.101 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.009 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.iscsi_port = 3260 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.010 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.iscsi_protocol = iscsi log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.010 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.iscsi_secondary_ip_addresses = [] log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.010 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.iscsi_target_flags = log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.011 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.iscsi_target_prefix = iqn.2010-10.org.openstack: log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.011 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.iscsi_write_cache = on log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.011 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.iser_helper = tgtadm log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.012 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.iser_ip_address = 10.0.0.101 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.012 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.iser_port = 3260 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.013 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.iser_target_prefix = iqn.2010-10.org.openstack: log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.013 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.max_over_subscription_ratio = 20.0 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.013 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.num_iser_scan_tries = 3 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.014 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.num_shell_tries = 3 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.014 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.num_volume_device_scan_tries = 3 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.014 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.rados_connect_timeout = -1 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.015 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.rados_connection_interval = 5 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.015 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.rados_connection_retries = 3 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.015 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.rbd_ceph_conf = log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.015 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.rbd_cluster_name = ceph log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.016 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.rbd_flatten_volume_from_snapshot = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.016 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.rbd_keyring_conf = log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.016 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.rbd_max_clone_depth = 5 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.017 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.rbd_pool = rbd log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.017 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.rbd_secret_uuid = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.017 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.rbd_store_chunk_size = 4 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.018 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.rbd_user = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.018 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.replication_connect_timeout = 5 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.018 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.replication_device = **** log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.019 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.report_discard_supported = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.019 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.report_dynamic_total_capacity = True log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.019 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.reserved_percentage = 0 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.020 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.scst_target_driver = iscsi log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.020 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.scst_target_iqn_name = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.020 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.storage_protocol = iscsi log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.021 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.suppress_requests_ssl_warnings = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.021 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.trace_flags = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.021 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.use_chap_auth = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.022 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.use_multipath_for_image_xfer = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.022 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.volume_backend_name = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.022 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.volume_clear = zero log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.023 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.volume_clear_ionice = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.023 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.volume_clear_size = 0 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.023 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.volume_copy_blkio_cgroup_name = cinder-volume-copy log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.024 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.volume_copy_bps_limit = 0 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.024 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.volume_dd_blocksize = 1M log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.024 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.025 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] backend_defaults.volumes_dir = /var/lib/cinder/volumes log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.025 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_concurrency.disable_process_locking = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.026 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_concurrency.lock_path = /var/lib/cinder/tmp log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.026 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] barbican.auth_endpoint = http://localhost/identity/v3 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.027 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] barbican.barbican_api_version = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.027 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] barbican.barbican_endpoint = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.027 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] barbican.number_of_retries = 60 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.028 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] barbican.retry_delay = 1 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.028 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] barbican.verify_ssl = True log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.028 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] keystone_authtoken.admin_password = **** log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.029 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] keystone_authtoken.admin_tenant_name = admin log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.029 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] keystone_authtoken.admin_token = **** log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.029 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] keystone_authtoken.admin_user = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.030 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] keystone_authtoken.auth_admin_prefix = log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.030 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] keystone_authtoken.auth_host = 127.0.0.1 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.031 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] keystone_authtoken.auth_port = 35357 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.031 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] keystone_authtoken.auth_protocol = http log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.031 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] keystone_authtoken.auth_section = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.032 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] keystone_authtoken.auth_type = password log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.032 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] keystone_authtoken.auth_uri = http://172.17.1.12:5000 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.032 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] keystone_authtoken.auth_version = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.033 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] keystone_authtoken.cache = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.033 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] keystone_authtoken.cafile = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.033 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] keystone_authtoken.certfile = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.034 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] keystone_authtoken.check_revocations_for_cached = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.034 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] keystone_authtoken.delay_auth_decision = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.034 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] keystone_authtoken.enforce_token_bind = permissive log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.035 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] keystone_authtoken.hash_algorithms = ['md5'] log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.035 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] keystone_authtoken.http_connect_timeout = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.036 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] keystone_authtoken.http_request_max_retries = 3 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.036 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] keystone_authtoken.identity_uri = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.036 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] keystone_authtoken.include_service_catalog = True log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.037 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] keystone_authtoken.insecure = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.037 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] keystone_authtoken.keyfile = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.037 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] keystone_authtoken.memcache_pool_conn_get_timeout = 10 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.038 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] keystone_authtoken.memcache_pool_dead_retry = 300 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.038 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] keystone_authtoken.memcache_pool_maxsize = 10 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.038 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] keystone_authtoken.memcache_pool_socket_timeout = 3 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.039 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] keystone_authtoken.memcache_pool_unused_timeout = 60 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.039 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] keystone_authtoken.memcache_secret_key = **** log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.039 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] keystone_authtoken.memcache_security_strategy = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.040 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] keystone_authtoken.memcache_use_advanced_pool = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.040 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] keystone_authtoken.memcached_servers = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.040 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] keystone_authtoken.region_name = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.041 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] keystone_authtoken.revocation_cache_time = 10 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.041 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] keystone_authtoken.service_token_roles = ['service'] log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.041 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] keystone_authtoken.service_token_roles_required = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.042 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] keystone_authtoken.signing_dir = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.042 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] keystone_authtoken.token_cache_time = 300 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.042 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_versionedobjects.fatal_exception_format_errors = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.043 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_policy.policy_default_rule = default log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.043 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_policy.policy_dirs = ['policy.d'] log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.044 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] oslo_policy.policy_file = /etc/cinder/policy.json log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.044 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] key_manager.api_class = cinder.keymgr.conf_key_mgr.ConfKeyManager log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.044 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] key_manager.fixed_key = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2887 >2018-06-24 13:00:25.045 139116 DEBUG oslo_service.service [req-563ade2f-0efd-41b2-a14f-d9dd6969b919 - - - - -] ******************************************************************************** log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2889 >2018-06-24 13:00:25.076 139328 DEBUG cinder.volume.driver [req-5412b44b-7b69-4daa-a560-9093d875a059 - - - - -] Initialized capabilities list: {'vendor_name': 'Open Source', 'multiattach': False, 'provisioned_capacity_gb': 0.0, 'volume_backend_name': 'tripleo_ceph', 'thin_provisioning_support': True, 'free_capacity_gb': 39.45, 'driver_version': '1.2.0', 'total_capacity_gb': 39.45, 'reserved_percentage': 0, 'max_over_subscription_ratio': 20.0, 'replication_enabled': False, 'properties': {'replication_enabled': {'type': 'boolean', 'description': u'Enables replication.', 'title': 'Replication'}, 'qos': {'type': 'boolean', 'description': u'Enables QoS.', 'title': 'QoS'}, 'compression': {'type': 'boolean', 'description': u'Enables compression.', 'title': 'Compression'}, 'thin_provisioning': {'type': 'boolean', 'description': u'Sets thin provisioning.', 'title': 'Thin Provisioning'}}, 'storage_protocol': 'ceph'}. init_capabilities /usr/lib/python2.7/site-packages/cinder/volume/driver.py:747 >2018-06-24 13:00:25.134 139328 DEBUG cinder.db.sqlalchemy.api [req-5412b44b-7b69-4daa-a560-9093d875a059 - - - - -] Building query based on filter _process_snaps_filters /usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py:3006 >2018-06-24 13:00:25.147 139328 DEBUG cinder.image.image_utils [req-5412b44b-7b69-4daa-a560-9093d875a059 - - - - -] Configuration image_conversion_dir is None or the path doesn't exist. cleanup_temporary_file /usr/lib/python2.7/site-packages/cinder/image/image_utils.py:568 >2018-06-24 13:00:25.148 139328 DEBUG cinder.volume.drivers.rbd [req-5412b44b-7b69-4daa-a560-9093d875a059 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:00:25.187 139328 DEBUG cinder.volume.drivers.rbd [req-5412b44b-7b69-4daa-a560-9093d875a059 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:00:25.226 139328 DEBUG cinder.manager [req-5412b44b-7b69-4daa-a560-9093d875a059 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:00:25.228 139328 WARNING oslo_config.cfg [req-5412b44b-7b69-4daa-a560-9093d875a059 - - - - -] Option "rabbit_port" from group "oslo_messaging_rabbit" is deprecated for removal (Replaced by [DEFAULT]/transport_url). Its value may be silently ignored in the future. >2018-06-24 13:00:25.229 139328 WARNING oslo_config.cfg [req-5412b44b-7b69-4daa-a560-9093d875a059 - - - - -] Option "rabbit_userid" from group "oslo_messaging_rabbit" is deprecated for removal (Replaced by [DEFAULT]/transport_url). Its value may be silently ignored in the future. >2018-06-24 13:00:25.229 139328 WARNING oslo_config.cfg [req-5412b44b-7b69-4daa-a560-9093d875a059 - - - - -] Option "rabbit_password" from group "oslo_messaging_rabbit" is deprecated for removal (Replaced by [DEFAULT]/transport_url). Its value may be silently ignored in the future. >2018-06-24 13:00:25.261 139328 INFO cinder.volume.manager [req-5412b44b-7b69-4daa-a560-9093d875a059 - - - - -] Driver initialization completed successfully. >2018-06-24 13:00:25.271 139328 INFO cinder.manager [req-5412b44b-7b69-4daa-a560-9093d875a059 - - - - -] Initiating service 5 cleanup >2018-06-24 13:00:25.276 139328 INFO cinder.manager [req-5412b44b-7b69-4daa-a560-9093d875a059 - - - - -] Service 5 cleanup completed. >2018-06-24 13:00:25.277 139328 DEBUG cinder.service [req-5412b44b-7b69-4daa-a560-9093d875a059 - - - - -] Creating RPC server for service cinder-volume start /usr/lib/python2.7/site-packages/cinder/service.py:219 >2018-06-24 13:00:25.282 139328 DEBUG cinder.service [req-5412b44b-7b69-4daa-a560-9093d875a059 - - - - -] Pinning object versions for RPC server serializer to 1.27 start /usr/lib/python2.7/site-packages/cinder/service.py:226 >2018-06-24 13:00:25.498 139328 INFO cinder.volume.manager [req-5412b44b-7b69-4daa-a560-9093d875a059 - - - - -] Initializing RPC dependent components of volume driver RBDDriver (1.2.0) >2018-06-24 13:00:25.499 139328 DEBUG cinder.volume.drivers.rbd [req-5412b44b-7b69-4daa-a560-9093d875a059 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:00:25.537 139328 DEBUG cinder.volume.drivers.rbd [req-5412b44b-7b69-4daa-a560-9093d875a059 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:00:25.596 139328 INFO cinder.volume.manager [req-5412b44b-7b69-4daa-a560-9093d875a059 - - - - -] Driver post RPC initialization completed successfully. >2018-06-24 13:01:39.597 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:01:39.599 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:01:39.606 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:01:39.607 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:01:39.649 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:02:39.599 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:02:39.600 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:02:39.608 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:02:39.609 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:02:39.648 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:03:39.601 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:03:39.603 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:03:39.613 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:03:39.614 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:03:39.658 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:04:39.597 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:04:39.598 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:04:39.607 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:04:39.608 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:04:39.652 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:05:39.596 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:05:39.597 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:05:39.607 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:05:39.608 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:05:39.650 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:06:39.592 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:06:39.595 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:06:39.604 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:06:39.605 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:06:39.649 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:07:39.588 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:07:39.590 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:07:39.597 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:07:39.598 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:07:39.647 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:08:39.587 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:08:39.588 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:08:39.597 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:08:39.597 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:08:39.638 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:09:39.588 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:09:39.590 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:09:39.598 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:09:39.599 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:09:39.647 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:10:21.284 139328 DEBUG cinder.volume.flows.manager.create_volume [req-4fc68c5e-e7af-4e91-9068-5930bdf34b81 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] Volume reschedule parameters: True retry: {u'num_attempts': 1, u'backends': [u'hostgroup@tripleo_ceph#tripleo_ceph'], u'hosts': [u'hostgroup@tripleo_ceph#tripleo_ceph']} get_flow /usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py:1075 >2018-06-24 13:10:21.347 139328 DEBUG cinder.volume.manager [req-4fc68c5e-e7af-4e91-9068-5930bdf34b81 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] Flow 'volume_create_manager' (7377aa6d-9e34-4d8a-8634-910179fc5de8) transitioned into state 'RUNNING' from state 'PENDING' _flow_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:145 >2018-06-24 13:10:21.350 139328 DEBUG cinder.volume.manager [req-4fc68c5e-e7af-4e91-9068-5930bdf34b81 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (95c4e9ea-fb48-474f-a774-d1f29e4be657) transitioned into state 'RUNNING' from state 'PENDING' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:194 >2018-06-24 13:10:21.372 139328 DEBUG oslo_db.sqlalchemy.engines [req-4fc68c5e-e7af-4e91-9068-5930bdf34b81 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:285 >2018-06-24 13:10:21.395 139328 DEBUG cinder.volume.manager [req-4fc68c5e-e7af-4e91-9068-5930bdf34b81 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (95c4e9ea-fb48-474f-a774-d1f29e4be657) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2018-06-24T13:10:20Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1039738307',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='hostgroup@tripleo_ceph#tripleo_ceph',id=f2100b53-5f96-4023-8394-7bb280bd65a6,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='dafc0bcea22b4e7c9db951f20c0a3ed1',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2018-06-24T13:10:21Z,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2018-06-24T13:10:21Z,user_id='4d2a2ed08fdd46599f3797d764614457',volume_attachment=<?>,volume_type=<?>,volume_type_id=None)' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:183 >2018-06-24 13:10:21.397 139328 DEBUG cinder.volume.manager [req-4fc68c5e-e7af-4e91-9068-5930bdf34b81 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (04080082-ea10-45dd-8d10-c497a7b34d3b) transitioned into state 'RUNNING' from state 'PENDING' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:194 >2018-06-24 13:10:21.399 139328 DEBUG cinder.volume.manager [req-4fc68c5e-e7af-4e91-9068-5930bdf34b81 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (04080082-ea10-45dd-8d10-c497a7b34d3b) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:183 >2018-06-24 13:10:21.401 139328 DEBUG cinder.volume.manager [req-4fc68c5e-e7af-4e91-9068-5930bdf34b81 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (70993cd4-19a9-450c-b6b2-07d0cf1d327a) transitioned into state 'RUNNING' from state 'PENDING' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:194 >2018-06-24 13:10:21.403 139328 DEBUG cinder.volume.manager [req-4fc68c5e-e7af-4e91-9068-5930bdf34b81 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (70993cd4-19a9-450c-b6b2-07d0cf1d327a) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': u'creating', 'volume_size': 1, 'volume_name': 'volume-f2100b53-5f96-4023-8394-7bb280bd65a6', 'type': 'raw', 'volume_id': 'f2100b53-5f96-4023-8394-7bb280bd65a6'}' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:183 >2018-06-24 13:10:21.404 139328 DEBUG cinder.volume.manager [req-4fc68c5e-e7af-4e91-9068-5930bdf34b81 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (f1c9f1b3-a942-4780-bd09-5327e29103e5) transitioned into state 'RUNNING' from state 'PENDING' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:194 >2018-06-24 13:10:21.471 139328 DEBUG cinder.volume.manager [req-4fc68c5e-e7af-4e91-9068-5930bdf34b81 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (f1c9f1b3-a942-4780-bd09-5327e29103e5) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:183 >2018-06-24 13:10:21.473 139328 DEBUG cinder.volume.manager [req-4fc68c5e-e7af-4e91-9068-5930bdf34b81 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (275e1d93-0099-4dbb-bfd8-130b3172fee7) transitioned into state 'RUNNING' from state 'PENDING' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:194 >2018-06-24 13:10:21.474 139328 INFO cinder.volume.flows.manager.create_volume [req-4fc68c5e-e7af-4e91-9068-5930bdf34b81 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] Volume f2100b53-5f96-4023-8394-7bb280bd65a6: being created as raw with specification: {'status': u'creating', 'volume_size': 1, 'volume_name': 'volume-f2100b53-5f96-4023-8394-7bb280bd65a6'} >2018-06-24 13:10:21.474 139328 DEBUG cinder.volume.drivers.rbd [req-4fc68c5e-e7af-4e91-9068-5930bdf34b81 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] creating volume 'volume-f2100b53-5f96-4023-8394-7bb280bd65a6' create_volume /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:655 >2018-06-24 13:10:21.475 139328 DEBUG cinder.volume.drivers.rbd [req-4fc68c5e-e7af-4e91-9068-5930bdf34b81 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:10:22.899 139328 DEBUG cinder.volume.manager [req-4fc68c5e-e7af-4e91-9068-5930bdf34b81 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (275e1d93-0099-4dbb-bfd8-130b3172fee7) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:183 >2018-06-24 13:10:22.901 139328 DEBUG cinder.volume.manager [req-4fc68c5e-e7af-4e91-9068-5930bdf34b81 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (ef4d966e-cba4-4e68-8d49-25954152bb1c) transitioned into state 'RUNNING' from state 'PENDING' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:194 >2018-06-24 13:10:22.930 139328 INFO cinder.volume.flows.manager.create_volume [req-4fc68c5e-e7af-4e91-9068-5930bdf34b81 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] Volume volume-f2100b53-5f96-4023-8394-7bb280bd65a6 (f2100b53-5f96-4023-8394-7bb280bd65a6): created successfully >2018-06-24 13:10:22.933 139328 DEBUG cinder.volume.manager [req-4fc68c5e-e7af-4e91-9068-5930bdf34b81 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (ef4d966e-cba4-4e68-8d49-25954152bb1c) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:183 >2018-06-24 13:10:22.937 139328 DEBUG cinder.volume.manager [req-4fc68c5e-e7af-4e91-9068-5930bdf34b81 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] Flow 'volume_create_manager' (7377aa6d-9e34-4d8a-8634-910179fc5de8) transitioned into state 'SUCCESS' from state 'RUNNING' _flow_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:145 >2018-06-24 13:10:22.937 139328 INFO cinder.volume.manager [req-4fc68c5e-e7af-4e91-9068-5930bdf34b81 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] Created volume successfully. >2018-06-24 13:10:35.049 139328 DEBUG oslo_concurrency.processutils [req-34cde1d7-0b06-4589-a680-079d66dd0e1f 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] Running cmd (subprocess): ceph mon dump --format=json --id openstack --cluster ceph --conf /etc/ceph/ceph.conf execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:367 >2018-06-24 13:10:35.405 139328 DEBUG oslo_concurrency.processutils [req-34cde1d7-0b06-4589-a680-079d66dd0e1f 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] CMD "ceph mon dump --format=json --id openstack --cluster ceph --conf /etc/ceph/ceph.conf" returned: 0 in 0.357s execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:404 >2018-06-24 13:10:35.407 139328 DEBUG cinder.volume.drivers.rbd [req-34cde1d7-0b06-4589-a680-079d66dd0e1f 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] connection data: {'driver_volume_type': 'rbd', 'data': {'secret_uuid': None, 'volume_id': 'f2100b53-5f96-4023-8394-7bb280bd65a6', 'auth_username': 'openstack', 'secret_type': 'ceph', 'name': 'volumes/volume-f2100b53-5f96-4023-8394-7bb280bd65a6', 'discard': True, 'keyring': None, 'cluster_name': 'ceph', 'hosts': [u'172.17.3.18', u'172.17.3.22', u'172.17.3.25'], 'auth_enabled': True, 'ports': [u'6789', u'6789', u'6789']}} initialize_connection /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:1136 >2018-06-24 13:10:35.407 139328 INFO cinder.volume.manager [req-34cde1d7-0b06-4589-a680-079d66dd0e1f 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] Initialize volume connection completed successfully. >2018-06-24 13:10:36.844 139328 DEBUG cinder.coordination [req-7042880e-9c0b-4219-aa16-8efc2f82c5d3 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] Lock "/var/lib/cinder/cinder-f2100b53-5f96-4023-8394-7bb280bd65a6" acquired by "attach_volume" :: waited 0.001s _synchronized /usr/lib/python2.7/site-packages/cinder/coordination.py:175 >2018-06-24 13:10:36.945 139328 INFO cinder.volume.manager [req-7042880e-9c0b-4219-aa16-8efc2f82c5d3 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] Attaching volume f2100b53-5f96-4023-8394-7bb280bd65a6 to instance 5bd5cba4-7580-49bf-ab56-d38c18405bf5 at mountpoint /dev/vdb on host None. >2018-06-24 13:10:37.036 139328 INFO cinder.volume.manager [req-7042880e-9c0b-4219-aa16-8efc2f82c5d3 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] Attach volume completed successfully. >2018-06-24 13:10:37.037 139328 DEBUG cinder.coordination [req-7042880e-9c0b-4219-aa16-8efc2f82c5d3 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] Lock "/var/lib/cinder/cinder-f2100b53-5f96-4023-8394-7bb280bd65a6" released by "attach_volume" :: held 0.193s _synchronized /usr/lib/python2.7/site-packages/cinder/coordination.py:187 >2018-06-24 13:10:39.590 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:10:39.591 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:10:39.598 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:10:39.598 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:10:39.636 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:10:39.672 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:11:18.783 139328 INFO cinder.volume.manager [req-2e00dbf7-bfce-40fe-aa17-fed611fb2e3d 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] Terminate volume connection completed successfully. >2018-06-24 13:11:20.055 139328 DEBUG cinder.coordination [req-d34e9f77-cb3d-4472-bb38-1a80e420b011 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] Lock "/var/lib/cinder/cinder-f2100b53-5f96-4023-8394-7bb280bd65a6-detach_volume" acquired by "detach_volume" :: waited 0.001s _synchronized /usr/lib/python2.7/site-packages/cinder/coordination.py:175 >2018-06-24 13:11:20.102 139328 INFO cinder.volume.manager [req-d34e9f77-cb3d-4472-bb38-1a80e420b011 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] Detaching volume f2100b53-5f96-4023-8394-7bb280bd65a6 from instance 5bd5cba4-7580-49bf-ab56-d38c18405bf5. >2018-06-24 13:11:20.232 139328 INFO cinder.volume.manager [req-d34e9f77-cb3d-4472-bb38-1a80e420b011 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] Detach volume completed successfully. >2018-06-24 13:11:20.233 139328 DEBUG cinder.coordination [req-d34e9f77-cb3d-4472-bb38-1a80e420b011 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] Lock "/var/lib/cinder/cinder-f2100b53-5f96-4023-8394-7bb280bd65a6-detach_volume" released by "detach_volume" :: held 0.178s _synchronized /usr/lib/python2.7/site-packages/cinder/coordination.py:187 >2018-06-24 13:11:20.630 139328 DEBUG cinder.coordination [req-1e485584-875b-4194-b7be-b1115757ee2a 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] Lock "/var/lib/cinder/cinder-f2100b53-5f96-4023-8394-7bb280bd65a6-delete_volume" acquired by "delete_volume" :: waited 0.000s _synchronized /usr/lib/python2.7/site-packages/cinder/coordination.py:175 >2018-06-24 13:11:20.907 139328 DEBUG cinder.volume.drivers.rbd [req-1e485584-875b-4194-b7be-b1115757ee2a 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:11:20.963 139328 DEBUG cinder.volume.drivers.rbd [req-1e485584-875b-4194-b7be-b1115757ee2a 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] volume has no backup snaps _delete_backup_snaps /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:736 >2018-06-24 13:11:20.964 139328 DEBUG cinder.volume.drivers.rbd [req-1e485584-875b-4194-b7be-b1115757ee2a 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] Volume volume-f2100b53-5f96-4023-8394-7bb280bd65a6 is not a clone. _get_clone_info /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:759 >2018-06-24 13:11:20.967 139328 DEBUG cinder.volume.drivers.rbd [req-1e485584-875b-4194-b7be-b1115757ee2a 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] deleting rbd volume volume-f2100b53-5f96-4023-8394-7bb280bd65a6 delete_volume /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:857 >2018-06-24 13:11:21.048 139328 DEBUG cinder.quota [req-1e485584-875b-4194-b7be-b1115757ee2a 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] Created reservations ['6a7cb364-91cb-4ed9-a2f0-bd8c53ce1954', '28a6572f-4422-474f-ae58-6bf3360a519d'] reserve /usr/lib/python2.7/site-packages/cinder/quota.py:1029 >2018-06-24 13:11:21.082 139328 DEBUG cinder.volume.utils [req-1e485584-875b-4194-b7be-b1115757ee2a 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] Can not find volume f2100b53-5f96-4023-8394-7bb280bd65a6 at notify usage _usage_from_volume /usr/lib/python2.7/site-packages/cinder/volume/utils.py:99 >2018-06-24 13:11:21.104 139328 DEBUG cinder.volume.drivers.rbd [req-1e485584-875b-4194-b7be-b1115757ee2a 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:11:21.147 139328 DEBUG cinder.volume.drivers.rbd [req-1e485584-875b-4194-b7be-b1115757ee2a 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:11:21.192 139328 DEBUG cinder.manager [req-1e485584-875b-4194-b7be-b1115757ee2a 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:11:21.201 139328 INFO cinder.volume.manager [req-1e485584-875b-4194-b7be-b1115757ee2a 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] Deleted volume successfully. >2018-06-24 13:11:21.207 139328 DEBUG cinder.coordination [req-1e485584-875b-4194-b7be-b1115757ee2a 4d2a2ed08fdd46599f3797d764614457 dafc0bcea22b4e7c9db951f20c0a3ed1 - default default] Lock "/var/lib/cinder/cinder-f2100b53-5f96-4023-8394-7bb280bd65a6-delete_volume" released by "delete_volume" :: held 0.577s _synchronized /usr/lib/python2.7/site-packages/cinder/coordination.py:187 >2018-06-24 13:11:39.595 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:11:39.596 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:12:39.594 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:12:39.595 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:12:39.602 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:12:39.603 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:12:39.644 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:13:39.593 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:13:39.594 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:13:39.606 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:13:39.607 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:13:39.645 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:14:39.595 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:14:39.597 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:14:39.606 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:14:39.606 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:14:39.648 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:15:39.599 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:15:39.600 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:15:39.638 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:16:39.597 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:16:39.598 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:16:39.607 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:16:39.607 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:16:39.647 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:17:39.595 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:17:39.596 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:17:39.606 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:17:39.607 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:17:39.653 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:18:39.595 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:18:39.596 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:18:39.604 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:18:39.604 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:18:39.643 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:19:39.594 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:19:39.595 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:19:39.603 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:19:39.604 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:19:39.646 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:20:39.596 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:20:39.597 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:20:39.604 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:20:39.605 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:20:39.644 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:21:39.593 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:21:39.595 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:21:39.605 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:21:39.606 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:21:39.646 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:22:39.597 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:22:39.599 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:22:39.606 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:22:39.606 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:22:39.647 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:23:39.597 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:23:39.599 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:23:39.609 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:23:39.610 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:23:39.666 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:24:39.599 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:24:39.601 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:24:39.609 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:24:39.610 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:24:39.651 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:25:39.603 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:25:39.604 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:25:39.612 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:25:39.612 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:25:39.659 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:26:39.600 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:26:39.601 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:26:39.609 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:26:39.609 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:26:39.654 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:27:39.597 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:27:39.599 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:27:39.606 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:27:39.607 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:27:39.645 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:28:39.594 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:28:39.595 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:28:39.604 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:28:39.604 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:28:39.643 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:29:39.593 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:29:39.595 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:29:39.602 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:29:39.603 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:29:39.657 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:30:39.594 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:30:39.595 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:30:39.604 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:30:39.605 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:30:39.647 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:31:39.594 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:31:39.596 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:31:39.604 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:31:39.605 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:31:39.647 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:32:39.597 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:32:39.599 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:32:39.606 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:32:39.607 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:32:39.648 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:33:39.597 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:33:39.598 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:33:39.605 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:33:39.605 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:33:39.644 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:34:39.595 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:34:39.596 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:34:39.603 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:34:39.604 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:34:39.654 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:35:39.594 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:35:39.596 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:35:39.604 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:35:39.605 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:35:39.653 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:36:39.594 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:36:39.596 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:36:39.604 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:36:39.605 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:36:39.646 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:37:39.596 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:37:39.598 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:37:39.612 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:37:39.613 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:37:39.659 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:38:39.597 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:38:39.599 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:38:39.607 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:38:39.608 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:38:39.653 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:39:39.600 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:39:39.602 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:39:39.615 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:39:39.616 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:39:39.666 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:40:39.602 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:40:39.604 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:40:39.614 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:40:39.615 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:40:39.664 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:41:39.602 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:41:39.604 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:41:39.611 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:41:39.612 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:41:39.668 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:42:39.603 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:42:39.605 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:42:39.613 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:42:39.613 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:42:39.658 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:43:39.602 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:43:39.604 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:43:39.610 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:43:39.611 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:43:39.654 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:44:39.606 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:44:39.609 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:44:39.618 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:44:39.619 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:44:39.668 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:45:39.607 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:45:39.609 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:45:39.617 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:45:39.617 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:45:39.656 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:45:39.656 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:45:39.656 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:45:39.656 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:45:39.656 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:45:39.656 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:45:39.656 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:45:39.662 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:45:39.662 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.044 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:45:39.662 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 13:45:49.663 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:45:49.699 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:45:49.699 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:45:49.699 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:45:49.699 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:45:49.699 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:45:49.699 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:45:49.699 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:45:49.702 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:45:49.702 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.084 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:45:49.702 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 13:46:09.703 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:46:09.741 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:46:09.741 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:46:09.741 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:46:09.741 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:46:09.741 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:46:09.741 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:46:09.741 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:46:09.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:46:09.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.126 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:46:09.745 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 13:46:09.745 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 13:46:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 13:46:09.745 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 13:46:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 13:46:09.745 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 13:46:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 13:46:09.745 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 13:46:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 13:46:09.745 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 13:46:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 13:46:09.745 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 13:46:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 13:46:09.745 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 13:46:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 13:46:09.745 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 13:46:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 13:46:09.745 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 13:46:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 13:46:09.745 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 13:46:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 13:46:09.745 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 13:46:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 13:46:09.745 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 13:46:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 13:46:09.745 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 13:46:09.745 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 13:46:09.745 139328 ERROR oslo_service.periodic_task >2018-06-24 13:46:39.603 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:46:39.605 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:46:39.614 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:46:39.614 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:46:39.656 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:46:39.656 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:46:39.656 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:46:39.656 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:46:39.656 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:46:39.656 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:46:39.656 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:46:39.659 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:46:39.659 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.045 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:46:39.660 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 13:46:49.660 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:46:49.699 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:46:49.699 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:46:49.699 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:46:49.699 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:46:49.699 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:46:49.699 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:46:49.699 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:46:49.702 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:46:49.703 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.088 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:46:49.703 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 13:47:09.704 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:47:09.742 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:47:09.742 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:47:09.742 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:47:09.742 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:47:09.742 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:47:09.742 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:47:09.742 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:47:09.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:47:09.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.129 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:47:09.744 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 13:47:09.744 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 13:47:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 13:47:09.744 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 13:47:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 13:47:09.744 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 13:47:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 13:47:09.744 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 13:47:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 13:47:09.744 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 13:47:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 13:47:09.744 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 13:47:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 13:47:09.744 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 13:47:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 13:47:09.744 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 13:47:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 13:47:09.744 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 13:47:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 13:47:09.744 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 13:47:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 13:47:09.744 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 13:47:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 13:47:09.744 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 13:47:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 13:47:09.744 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 13:47:09.744 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 13:47:09.744 139328 ERROR oslo_service.periodic_task >2018-06-24 13:47:39.606 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:47:39.608 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:47:39.620 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:47:39.621 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:47:39.658 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:47:39.658 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:47:39.658 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:47:39.658 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:47:39.658 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:47:39.658 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:47:39.658 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:47:39.661 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:47:39.662 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.041 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:47:39.662 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 13:47:49.662 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:47:49.697 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:47:49.697 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:47:49.697 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:47:49.697 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:47:49.697 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:47:49.697 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:47:49.697 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:47:49.700 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:47:49.700 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:47:49.700 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 13:48:09.701 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:48:09.749 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:48:09.749 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:48:09.749 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:48:09.749 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:48:09.749 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:48:09.749 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:48:09.749 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:48:09.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:48:09.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.131 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:48:09.753 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 13:48:09.753 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 13:48:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 13:48:09.753 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 13:48:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 13:48:09.753 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 13:48:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 13:48:09.753 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 13:48:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 13:48:09.753 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 13:48:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 13:48:09.753 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 13:48:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 13:48:09.753 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 13:48:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 13:48:09.753 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 13:48:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 13:48:09.753 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 13:48:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 13:48:09.753 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 13:48:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 13:48:09.753 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 13:48:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 13:48:09.753 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 13:48:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 13:48:09.753 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 13:48:09.753 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 13:48:09.753 139328 ERROR oslo_service.periodic_task >2018-06-24 13:48:39.604 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:48:39.606 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:48:39.615 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:48:39.616 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:48:39.654 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:48:39.654 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:48:39.654 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:48:39.654 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:48:39.654 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:48:39.654 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:48:39.654 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:48:39.658 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:48:39.659 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.042 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:48:39.659 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 13:48:49.660 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:48:49.694 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:48:49.694 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:48:49.694 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:48:49.694 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:48:49.694 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:48:49.694 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:48:49.694 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:48:49.697 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:48:49.697 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.08 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:48:49.697 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 13:49:09.698 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:49:09.738 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:49:09.738 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:49:09.738 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:49:09.738 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:49:09.738 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:49:09.738 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:49:09.738 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:49:09.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:49:09.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.123 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:49:09.741 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 13:49:09.741 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 13:49:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 13:49:09.741 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 13:49:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 13:49:09.741 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 13:49:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 13:49:09.741 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 13:49:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 13:49:09.741 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 13:49:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 13:49:09.741 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 13:49:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 13:49:09.741 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 13:49:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 13:49:09.741 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 13:49:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 13:49:09.741 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 13:49:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 13:49:09.741 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 13:49:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 13:49:09.741 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 13:49:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 13:49:09.741 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 13:49:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 13:49:09.741 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 13:49:09.741 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 13:49:09.741 139328 ERROR oslo_service.periodic_task >2018-06-24 13:49:39.603 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:49:39.605 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:49:39.614 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:49:39.615 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:49:39.655 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:49:39.655 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:49:39.655 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:49:39.655 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:49:39.655 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:49:39.655 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:49:39.655 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:49:39.658 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:49:39.658 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.042 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:49:39.658 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 13:49:49.659 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:49:49.691 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:49:49.691 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:49:49.691 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:49:49.691 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:49:49.691 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:49:49.691 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:49:49.691 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:49:49.693 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:49:49.694 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:49:49.694 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 13:50:09.694 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:50:09.732 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:50:09.732 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:50:09.732 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:50:09.732 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:50:09.732 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:50:09.732 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:50:09.732 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:50:09.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:50:09.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.119 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:50:09.735 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 13:50:09.735 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 13:50:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 13:50:09.735 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 13:50:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 13:50:09.735 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 13:50:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 13:50:09.735 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 13:50:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 13:50:09.735 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 13:50:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 13:50:09.735 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 13:50:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 13:50:09.735 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 13:50:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 13:50:09.735 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 13:50:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 13:50:09.735 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 13:50:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 13:50:09.735 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 13:50:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 13:50:09.735 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 13:50:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 13:50:09.735 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 13:50:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 13:50:09.735 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 13:50:09.735 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 13:50:09.735 139328 ERROR oslo_service.periodic_task >2018-06-24 13:50:39.607 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:50:39.608 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:50:39.616 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:50:39.617 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:50:39.650 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:50:39.650 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:50:39.650 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:50:39.650 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:50:39.650 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:50:39.650 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:50:39.650 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:50:39.653 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:50:39.653 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:50:39.653 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 13:50:49.654 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:50:49.694 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:50:49.694 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:50:49.694 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:50:49.694 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:50:49.694 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:50:49.694 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:50:49.694 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:50:49.696 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:50:49.697 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.08 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:50:49.697 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 13:51:09.698 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:51:09.735 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:51:09.735 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:51:09.735 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:51:09.735 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:51:09.735 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:51:09.735 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:51:09.735 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:51:09.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:51:09.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.12 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:51:09.738 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 13:51:09.738 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 13:51:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 13:51:09.738 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 13:51:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 13:51:09.738 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 13:51:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 13:51:09.738 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 13:51:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 13:51:09.738 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 13:51:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 13:51:09.738 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 13:51:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 13:51:09.738 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 13:51:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 13:51:09.738 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 13:51:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 13:51:09.738 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 13:51:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 13:51:09.738 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 13:51:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 13:51:09.738 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 13:51:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 13:51:09.738 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 13:51:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 13:51:09.738 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 13:51:09.738 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 13:51:09.738 139328 ERROR oslo_service.periodic_task >2018-06-24 13:51:39.609 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:51:39.611 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:51:39.621 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:51:39.622 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:51:39.658 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:51:39.658 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:51:39.658 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:51:39.658 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:51:39.658 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:51:39.658 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:51:39.658 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:51:39.662 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:51:39.662 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:51:39.663 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 13:51:49.663 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:51:49.702 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:51:49.702 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:51:49.702 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:51:49.702 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:51:49.702 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:51:49.702 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:51:49.702 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:51:49.705 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:51:49.705 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.083 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:51:49.706 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 13:52:09.706 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:52:09.744 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:52:09.744 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:52:09.744 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:52:09.744 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:52:09.744 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:52:09.744 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:52:09.744 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:52:09.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:52:09.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.125 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:52:09.747 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 13:52:09.747 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 13:52:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 13:52:09.747 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 13:52:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 13:52:09.747 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 13:52:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 13:52:09.747 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 13:52:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 13:52:09.747 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 13:52:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 13:52:09.747 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 13:52:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 13:52:09.747 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 13:52:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 13:52:09.747 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 13:52:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 13:52:09.747 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 13:52:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 13:52:09.747 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 13:52:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 13:52:09.747 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 13:52:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 13:52:09.747 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 13:52:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 13:52:09.747 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 13:52:09.747 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 13:52:09.747 139328 ERROR oslo_service.periodic_task >2018-06-24 13:52:39.609 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:52:39.611 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:52:39.619 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:52:39.619 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:52:39.661 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:52:39.661 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:52:39.661 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:52:39.661 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:52:39.661 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:52:39.661 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:52:39.661 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:52:39.664 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:52:39.664 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.045 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:52:39.665 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 13:52:49.665 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:52:49.726 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:52:49.726 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:52:49.726 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:52:49.726 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:52:49.726 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:52:49.726 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:52:49.726 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:52:49.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:52:49.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:52:49.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 13:53:09.731 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:53:09.828 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:53:09.828 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:53:09.828 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:53:09.828 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:53:09.828 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:53:09.828 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:53:09.828 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:53:09.836 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:53:09.836 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.216 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:53:09.836 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 13:53:09.836 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 13:53:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 13:53:09.836 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 13:53:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 13:53:09.836 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 13:53:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 13:53:09.836 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 13:53:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 13:53:09.836 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 13:53:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 13:53:09.836 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 13:53:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 13:53:09.836 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 13:53:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 13:53:09.836 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 13:53:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 13:53:09.836 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 13:53:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 13:53:09.836 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 13:53:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 13:53:09.836 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 13:53:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 13:53:09.836 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 13:53:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 13:53:09.836 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 13:53:09.836 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 13:53:09.836 139328 ERROR oslo_service.periodic_task >2018-06-24 13:53:39.608 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:53:39.610 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:53:39.617 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:53:39.618 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:53:39.655 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:53:39.655 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:53:39.655 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:53:39.655 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:53:39.655 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:53:39.655 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:53:39.655 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:53:39.658 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:53:39.659 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.041 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:53:39.659 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 13:53:49.660 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:53:49.712 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:53:49.712 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:53:49.712 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:53:49.712 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:53:49.712 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:53:49.712 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:53:49.712 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:53:49.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:53:49.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.098 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:53:49.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 13:54:09.717 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:54:09.757 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:54:09.757 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:54:09.757 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:54:09.757 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:54:09.757 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:54:09.757 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:54:09.757 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:54:09.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:54:09.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.142 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:54:09.761 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 13:54:09.761 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 13:54:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 13:54:09.761 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 13:54:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 13:54:09.761 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 13:54:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 13:54:09.761 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 13:54:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 13:54:09.761 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 13:54:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 13:54:09.761 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 13:54:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 13:54:09.761 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 13:54:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 13:54:09.761 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 13:54:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 13:54:09.761 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 13:54:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 13:54:09.761 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 13:54:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 13:54:09.761 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 13:54:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 13:54:09.761 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 13:54:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 13:54:09.761 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 13:54:09.761 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 13:54:09.761 139328 ERROR oslo_service.periodic_task >2018-06-24 13:54:39.612 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:54:39.613 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:54:39.626 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:54:39.627 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:54:39.667 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:54:39.667 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:54:39.667 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:54:39.667 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:54:39.667 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:54:39.667 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:54:39.667 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:54:39.669 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:54:39.670 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.043 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:54:39.670 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 13:54:49.671 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:54:49.709 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:54:49.709 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:54:49.709 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:54:49.709 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:54:49.709 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:54:49.709 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:54:49.709 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:54:49.711 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:54:49.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.085 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:54:49.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 13:55:09.712 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:55:09.752 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:55:09.752 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:55:09.752 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:55:09.752 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:55:09.752 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:55:09.752 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:55:09.752 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:55:09.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:55:09.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.128 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:55:09.755 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 13:55:09.755 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 13:55:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 13:55:09.755 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 13:55:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 13:55:09.755 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 13:55:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 13:55:09.755 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 13:55:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 13:55:09.755 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 13:55:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 13:55:09.755 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 13:55:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 13:55:09.755 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 13:55:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 13:55:09.755 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 13:55:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 13:55:09.755 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 13:55:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 13:55:09.755 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 13:55:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 13:55:09.755 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 13:55:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 13:55:09.755 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 13:55:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 13:55:09.755 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 13:55:09.755 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 13:55:09.755 139328 ERROR oslo_service.periodic_task >2018-06-24 13:55:39.617 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:55:39.619 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:55:39.626 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:55:39.627 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:55:39.665 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:55:39.665 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:55:39.665 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:55:39.665 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:55:39.665 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:55:39.665 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:55:39.665 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:55:39.667 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:55:39.667 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:55:39.668 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 13:55:49.668 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:55:49.702 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:55:49.702 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:55:49.702 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:55:49.702 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:55:49.702 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:55:49.702 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:55:49.702 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:55:49.704 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:55:49.704 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:55:49.704 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 13:56:09.705 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:56:09.748 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:56:09.748 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:56:09.748 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:56:09.748 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:56:09.748 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:56:09.748 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:56:09.748 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:56:09.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:56:09.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.124 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:56:09.751 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 13:56:09.751 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 13:56:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 13:56:09.751 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 13:56:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 13:56:09.751 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 13:56:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 13:56:09.751 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 13:56:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 13:56:09.751 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 13:56:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 13:56:09.751 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 13:56:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 13:56:09.751 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 13:56:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 13:56:09.751 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 13:56:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 13:56:09.751 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 13:56:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 13:56:09.751 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 13:56:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 13:56:09.751 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 13:56:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 13:56:09.751 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 13:56:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 13:56:09.751 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 13:56:09.751 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 13:56:09.751 139328 ERROR oslo_service.periodic_task >2018-06-24 13:56:39.613 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:56:39.614 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:56:39.621 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:56:39.622 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:56:39.661 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:56:39.661 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:56:39.661 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:56:39.661 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:56:39.661 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:56:39.661 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:56:39.661 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:56:39.663 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:56:39.664 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.041 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:56:39.664 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 13:56:49.665 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:56:49.699 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:56:49.699 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:56:49.699 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:56:49.699 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:56:49.699 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:56:49.699 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:56:49.699 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:56:49.701 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:56:49.702 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:56:49.702 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 13:57:09.702 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:57:09.738 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:57:09.738 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:57:09.738 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:57:09.738 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:57:09.738 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:57:09.738 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:57:09.738 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:57:09.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:57:09.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.118 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:57:09.742 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 13:57:09.742 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 13:57:09.742 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 13:57:09.742 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 13:57:09.742 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 13:57:09.742 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 13:57:09.742 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 13:57:09.742 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 13:57:09.742 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 13:57:09.742 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 13:57:09.742 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 13:57:09.742 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 13:57:09.742 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 13:57:09.742 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 13:57:09.742 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 13:57:09.742 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 13:57:09.742 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 13:57:09.742 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 13:57:09.742 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 13:57:09.742 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 13:57:09.742 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 13:57:09.742 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 13:57:09.742 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 13:57:09.742 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 13:57:09.742 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 13:57:09.742 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 13:57:09.742 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 13:57:09.742 139328 ERROR oslo_service.periodic_task >2018-06-24 13:57:39.613 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:57:39.614 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:57:39.621 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:57:39.621 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:57:39.660 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:57:39.660 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:57:39.660 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:57:39.660 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:57:39.660 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:57:39.660 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:57:39.660 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:57:39.662 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:57:39.663 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.041 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:57:39.663 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 13:57:49.663 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:57:49.699 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:57:49.699 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:57:49.699 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:57:49.699 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:57:49.699 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:57:49.699 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:57:49.699 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:57:49.701 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:57:49.702 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.08 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:57:49.702 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 13:58:09.703 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:58:09.738 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:58:09.738 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:58:09.738 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:58:09.738 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:58:09.738 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:58:09.738 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:58:09.738 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:58:09.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:58:09.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.119 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:58:09.741 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 13:58:09.741 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 13:58:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 13:58:09.741 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 13:58:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 13:58:09.741 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 13:58:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 13:58:09.741 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 13:58:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 13:58:09.741 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 13:58:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 13:58:09.741 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 13:58:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 13:58:09.741 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 13:58:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 13:58:09.741 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 13:58:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 13:58:09.741 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 13:58:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 13:58:09.741 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 13:58:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 13:58:09.741 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 13:58:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 13:58:09.741 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 13:58:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 13:58:09.741 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 13:58:09.741 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 13:58:09.741 139328 ERROR oslo_service.periodic_task >2018-06-24 13:58:39.613 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:58:39.615 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:58:39.624 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:58:39.624 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:58:39.668 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:58:39.668 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:58:39.668 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:58:39.668 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:58:39.668 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:58:39.668 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:58:39.668 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:58:39.672 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:58:39.672 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.047 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:58:39.672 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 13:58:49.673 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:58:49.729 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:58:49.729 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:58:49.729 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:58:49.729 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:58:49.729 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:58:49.729 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:58:49.729 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:58:49.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:58:49.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.106 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:58:49.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 13:59:08.339 139328 DEBUG cinder.volume.flows.manager.create_volume [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Volume reschedule parameters: True retry: {u'num_attempts': 1, u'backends': [u'hostgroup@tripleo_ceph#tripleo_ceph'], u'hosts': [u'hostgroup@tripleo_ceph#tripleo_ceph']} get_flow /usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py:1075 >2018-06-24 13:59:08.358 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Flow 'volume_create_manager' (147bc965-d641-428a-b91a-8d234430a694) transitioned into state 'RUNNING' from state 'PENDING' _flow_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:145 >2018-06-24 13:59:08.361 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (96a893cc-7f92-4fe5-977c-439b6350a172) transitioned into state 'RUNNING' from state 'PENDING' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:194 >2018-06-24 13:59:08.382 139328 DEBUG oslo_db.sqlalchemy.engines [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:285 >2018-06-24 13:59:08.408 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (96a893cc-7f92-4fe5-977c-439b6350a172) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2018-06-24T13:59:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1056713959',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='hostgroup@tripleo_ceph#tripleo_ceph',id=b29f1542-c089-463c-8354-fa3d8de9030e,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='3ee2014cc6684c8484b6a20dcabefd29',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2018-06-24T13:59:08Z,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2018-06-24T13:59:08Z,user_id='239cfa9cffab4bc89f7b738ebd9f59c8',volume_attachment=<?>,volume_type=<?>,volume_type_id=None)' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:183 >2018-06-24 13:59:08.411 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (8ad5fcce-9c37-43d1-8ae9-6922efda7b9b) transitioned into state 'RUNNING' from state 'PENDING' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:194 >2018-06-24 13:59:08.413 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (8ad5fcce-9c37-43d1-8ae9-6922efda7b9b) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:183 >2018-06-24 13:59:08.415 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a2504172-c94f-4088-a643-e6e821879898) transitioned into state 'RUNNING' from state 'PENDING' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:194 >2018-06-24 13:59:08.417 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a2504172-c94f-4088-a643-e6e821879898) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': u'creating', 'volume_size': 1, 'volume_name': 'volume-b29f1542-c089-463c-8354-fa3d8de9030e', 'type': 'raw', 'volume_id': 'b29f1542-c089-463c-8354-fa3d8de9030e'}' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:183 >2018-06-24 13:59:08.419 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (da86c43e-1200-40e5-8081-89a4bba18273) transitioned into state 'RUNNING' from state 'PENDING' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:194 >2018-06-24 13:59:08.446 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (da86c43e-1200-40e5-8081-89a4bba18273) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:183 >2018-06-24 13:59:08.449 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (86e72693-8c49-4932-83f5-cafa96616c8a) transitioned into state 'RUNNING' from state 'PENDING' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:194 >2018-06-24 13:59:08.449 139328 INFO cinder.volume.flows.manager.create_volume [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Volume b29f1542-c089-463c-8354-fa3d8de9030e: being created as raw with specification: {'status': u'creating', 'volume_size': 1, 'volume_name': 'volume-b29f1542-c089-463c-8354-fa3d8de9030e'} >2018-06-24 13:59:08.450 139328 DEBUG cinder.volume.drivers.rbd [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] creating volume 'volume-b29f1542-c089-463c-8354-fa3d8de9030e' create_volume /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:655 >2018-06-24 13:59:08.451 139328 DEBUG cinder.volume.drivers.rbd [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:59:08.485 139328 ERROR cinder.volume.drivers.rbd [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:59:08.485 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:59:08.485 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:59:08.485 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:59:08.485 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:59:08.485 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:59:08.485 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:59:08.488 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:59:08.489 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:59:08.490 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 13:59:09.732 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:59:09.768 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:59:09.768 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:59:09.768 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:59:09.768 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:59:09.768 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:59:09.768 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:59:09.768 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:59:09.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:59:09.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.145 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:59:09.770 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 13:59:09.770 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 13:59:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 13:59:09.770 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 13:59:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 13:59:09.770 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 13:59:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 13:59:09.770 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 13:59:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 13:59:09.770 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 13:59:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 13:59:09.770 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 13:59:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 13:59:09.770 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 13:59:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 13:59:09.770 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 13:59:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 13:59:09.770 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 13:59:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 13:59:09.770 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 13:59:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 13:59:09.770 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 13:59:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 13:59:09.770 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 13:59:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 13:59:09.770 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 13:59:09.770 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 13:59:09.770 139328 ERROR oslo_service.periodic_task >2018-06-24 13:59:18.490 139328 DEBUG cinder.volume.drivers.rbd [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:59:18.525 139328 ERROR cinder.volume.drivers.rbd [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:59:18.525 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:59:18.525 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:59:18.525 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:59:18.525 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:59:18.525 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:59:18.525 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:59:18.527 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:59:18.527 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:59:18.527 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 13:59:38.528 139328 DEBUG cinder.volume.drivers.rbd [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:59:38.561 139328 ERROR cinder.volume.drivers.rbd [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:59:38.561 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:59:38.561 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:59:38.561 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:59:38.561 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:59:38.561 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:59:38.561 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:59:38.564 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:59:38.564 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:59:38.573 139328 WARNING cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (86e72693-8c49-4932-83f5-cafa96616c8a) transitioned into state 'FAILURE' from state 'RUNNING' >5 predecessors (most recent first): > Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'volume': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2018-06-24T13:59:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1056713959',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='hostgroup@tripleo_ceph#tripleo_ceph',id=b29f1542-c089-463c-8354-fa3d8de9030e,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='3ee2014cc6684c8484b6a20dcabefd29',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2018-06-24T13:59:08Z,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2018-06-24T13:59:08Z,user_id='239cfa9cffab4bc89f7b738ebd9f59c8',volume_attachment=<?>,volume_type=<?>,volume_type_id=None), 'context': <cinder.context.RequestContext object at 0x7f796f9451d0>}, 'provides': None} > |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'volume': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2018-06-24T13:59:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1056713959',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='hostgroup@tripleo_ceph#tripleo_ceph',id=b29f1542-c089-463c-8354-fa3d8de9030e,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='3ee2014cc6684c8484b6a20dcabefd29',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2018-06-24T13:59:08Z,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2018-06-24T13:59:08Z,user_id='239cfa9cffab4bc89f7b738ebd9f59c8',volume_attachment=<?>,volume_type=<?>,volume_type_id=None), 'request_spec': RequestSpec(CG_backend=<?>,cgsnapshot_id=None,consistencygroup_id=None,group_backend=<?>,group_id=None,image_id=None,snapshot_id=None,source_replicaid=None,source_volid=None,volume=Volume(b29f1542-c089-463c-8354-fa3d8de9030e),volume_id=b29f1542-c089-463c-8354-fa3d8de9030e,volume_properties=VolumeProperties,volume_type=None), 'context': <cinder.context.RequestContext object at 0x7f796f9451d0>}, 'provides': {'status': u'creating', 'volume_size': 1, 'volume_name': 'volume-b29f1542-c089-463c-8354-fa3d8de9030e', 'type': 'raw', 'volume_id': 'b29f1542-c089-463c-8354-fa3d8de9030e'}} > |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'volume': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2018-06-24T13:59:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1056713959',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='hostgroup@tripleo_ceph#tripleo_ceph',id=b29f1542-c089-463c-8354-fa3d8de9030e,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='3ee2014cc6684c8484b6a20dcabefd29',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2018-06-24T13:59:08Z,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2018-06-24T13:59:08Z,user_id='239cfa9cffab4bc89f7b738ebd9f59c8',volume_attachment=<?>,volume_type=<?>,volume_type_id=None), 'filter_properties': {u'config_options': {}, u'request_spec': {u'resource_properties': {u'status': u'creating', u'volume_type_id': None, u'group_id': None, u'user_id': u'239cfa9cffab4bc89f7b738ebd9f59c8', u'availability_zone': u'nova', u'reservations': [u'0e00f6b6-7527-4b5a-9184-b8ac9ffaa4d3', u'b4bce244-a3c7-4874-9e57-3e423c8bc6c6'], u'multiattach': False, u'attach_status': u'detached', u'source_volid': None, u'cgsnapshot_id': None, u'metadata': {}, u'qos_specs': None, u'encryption_key_id': None, u'display_description': None, u'snapshot_id': None, u'source_replicaid': None, u'project_id': u'3ee2014cc6684c8484b6a20dcabefd29', u'consistencygroup_id': None, u'size': 1, u'display_name': u'tempest-TestMinimumBasicScenario-volume-1056713959'}, u'source_replicaid': None, u'volume_properties': {u'status': u'creating', u'volume_type_id': None, u'group_id': None, u'user_id': u'239cfa9cffab4bc89f7b738ebd9f59c8', u'availability_zone': u'nova', u'reservations': [u'0e00f6b6-7527-4b5a-9184-b8ac9ffaa4d3', u'b4bce244-a3c7-4874-9e57-3e423c8bc6c6'], u'multiattach': False, u'attach_status': u'detached', u'source_volid': None, u'cgsnapshot_id': None, u'metadata': {}, u'qos_specs': None, u'encryption_key_id': None, u'display_description': None, u'snapshot_id': None, u'source_replicaid': None, u'project_id': u'3ee2014cc6684c8484b6a20dcabefd29', u'consistencygroup_id': None, u'size': 1, u'display_name': u'tempest-TestMinimumBasicScenario-volume-1056713959'}, u'source_volid': None, u'cgsnapshot_id': None, u'volume': {u'migration_status': None, u'provider_id': None, u'availability_zone': u'nova', u'terminated_at': None, u'updated_at': None, u'provider_geometry': None, u'replication_extended_status': None, u'replication_status': None, u'snapshot_id': None, u'ec2_id': None, u'deleted_at': None, u'id': u'b29f1542-c089-463c-8354-fa3d8de9030e', u'size': 1, u'display_name': u'tempest-TestMinimumBasicScenario-volume-1056713959', u'display_description': None, u'cluster_name': None, u'name_id': u'b29f1542-c089-463c-8354-fa3d8de9030e', u'volume_admin_metadata': [], u'project_id': u'3ee2014cc6684c8484b6a20dcabefd29', u'launched_at': None, u'scheduled_at': None, u'status': u'creating', u'volume_type_id': None, u'multiattach': False, u'deleted': False, u'provider_location': None, u'volume_glance_metadata': [], u'host': None, u'consistencygroup_id': None, u'source_volid': None, u'provider_auth': None, u'previous_status': None, u'group_id': None, u'name': u'volume-b29f1542-c089-463c-8354-fa3d8de9030e', u'user_id': u'239cfa9cffab4bc89f7b738ebd9f59c8', u'bootable': False, u'created_at': u'2018-06-24T13:59:08.000000', u'attach_status': u'detached', u'volume_metadata': [], u'_name_id': None, u'encryption_key_id': None, u'replication_driver_data': None, u'metadata': {}}, u'image_id': None, u'snapshot_id': None, u'consistencygroup_id': None, u'volume_type': None, u'volume_id': u'b29f1542-c089-463c-8354-fa3d8de9030e', u'group_id': None}, u'user_id': u'239cfa9cffab4bc89f7b738ebd9f59c8', u'availability_zone': u'nova', u'volume_type': None, u'qos_specs': None, u'retry': {u'num_attempts': 1, u'backends': [u'hostgroup@tripleo_ceph#tripleo_ceph'], u'hosts': [u'hostgroup@tripleo_ceph#tripleo_ceph']}, u'metadata': {}, u'resource_type': {}, u'size': 1}, 'context': <cinder.context.RequestContext object at 0x7f796f9451d0>, 'request_spec': RequestSpec(CG_backend=<?>,cgsnapshot_id=None,consistencygroup_id=None,group_backend=<?>,group_id=None,image_id=None,snapshot_id=None,source_replicaid=None,source_volid=None,volume=Volume(b29f1542-c089-463c-8354-fa3d8de9030e),volume_id=b29f1542-c089-463c-8354-fa3d8de9030e,volume_properties=VolumeProperties,volume_type=None)}, 'provides': None} > |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'volume': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2018-06-24T13:59:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1056713959',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='hostgroup@tripleo_ceph#tripleo_ceph',id=b29f1542-c089-463c-8354-fa3d8de9030e,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='3ee2014cc6684c8484b6a20dcabefd29',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2018-06-24T13:59:08Z,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2018-06-24T13:59:08Z,user_id='239cfa9cffab4bc89f7b738ebd9f59c8',volume_attachment=<?>,volume_type=<?>,volume_type_id=None), 'context': <cinder.context.RequestContext object at 0x7f796f9451d0>}, 'provides': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2018-06-24T13:59:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1056713959',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='hostgroup@tripleo_ceph#tripleo_ceph',id=b29f1542-c089-463c-8354-fa3d8de9030e,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='3ee2014cc6684c8484b6a20dcabefd29',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2018-06-24T13:59:08Z,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2018-06-24T13:59:08Z,user_id='239cfa9cffab4bc89f7b738ebd9f59c8',volume_attachment=<?>,volume_type=<?>,volume_type_id=None)} > |__Flow 'volume_create_manager': VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 13:59:38.573 139328 ERROR cinder.volume.manager Traceback (most recent call last): >2018-06-24 13:59:38.573 139328 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task >2018-06-24 13:59:38.573 139328 ERROR cinder.volume.manager result = task.execute(**arguments) >2018-06-24 13:59:38.573 139328 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 942, in execute >2018-06-24 13:59:38.573 139328 ERROR cinder.volume.manager model_update = self._create_raw_volume(volume, **volume_spec) >2018-06-24 13:59:38.573 139328 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 910, in _create_raw_volume >2018-06-24 13:59:38.573 139328 ERROR cinder.volume.manager ret = self.driver.create_volume(volume) >2018-06-24 13:59:38.573 139328 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 661, in create_volume >2018-06-24 13:59:38.573 139328 ERROR cinder.volume.manager with RADOSClient(self) as client: >2018-06-24 13:59:38.573 139328 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 13:59:38.573 139328 ERROR cinder.volume.manager self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 13:59:38.573 139328 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 13:59:38.573 139328 ERROR cinder.volume.manager return _do_conn(pool, remote, timeout) >2018-06-24 13:59:38.573 139328 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 13:59:38.573 139328 ERROR cinder.volume.manager return r.call(f, *args, **kwargs) >2018-06-24 13:59:38.573 139328 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 13:59:38.573 139328 ERROR cinder.volume.manager raise attempt.get() >2018-06-24 13:59:38.573 139328 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 13:59:38.573 139328 ERROR cinder.volume.manager six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 13:59:38.573 139328 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 13:59:38.573 139328 ERROR cinder.volume.manager attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 13:59:38.573 139328 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 13:59:38.573 139328 ERROR cinder.volume.manager raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 13:59:38.573 139328 ERROR cinder.volume.manager VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 13:59:38.573 139328 ERROR cinder.volume.manager >2018-06-24 13:59:38.578 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (86e72693-8c49-4932-83f5-cafa96616c8a) transitioned into state 'REVERTING' from state 'FAILURE' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:194 >2018-06-24 13:59:38.580 139328 WARNING cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (86e72693-8c49-4932-83f5-cafa96616c8a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' >2018-06-24 13:59:38.582 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (da86c43e-1200-40e5-8081-89a4bba18273) transitioned into state 'REVERTING' from state 'SUCCESS' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:194 >2018-06-24 13:59:38.583 139328 WARNING cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (da86c43e-1200-40e5-8081-89a4bba18273) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' >2018-06-24 13:59:38.585 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a2504172-c94f-4088-a643-e6e821879898) transitioned into state 'REVERTING' from state 'SUCCESS' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:194 >2018-06-24 13:59:38.587 139328 WARNING cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a2504172-c94f-4088-a643-e6e821879898) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' >2018-06-24 13:59:38.589 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (8ad5fcce-9c37-43d1-8ae9-6922efda7b9b) transitioned into state 'REVERTING' from state 'SUCCESS' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:194 >2018-06-24 13:59:38.590 139328 DEBUG cinder.volume.flows.manager.create_volume [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Updating volume b29f1542-c089-463c-8354-fa3d8de9030e with {'host': None, 'scheduled_at': datetime.datetime(2018, 6, 24, 13, 59, 38, 589964)}. _pre_reschedule /usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py:121 >2018-06-24 13:59:38.597 139328 DEBUG cinder.volume.flows.manager.create_volume [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Volume b29f1542-c089-463c-8354-fa3d8de9030e: re-scheduling SchedulerAPI.create_volume attempt 1 due to Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. _reschedule /usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py:148 >2018-06-24 13:59:38.606 139328 INFO cinder.rpc [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Automatically selected cinder-volume objects version 1.27 as minimum service version. >2018-06-24 13:59:38.626 139328 DEBUG cinder.volume.flows.manager.create_volume [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Volume b29f1542-c089-463c-8354-fa3d8de9030e: re-scheduled _post_reschedule /usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py:160 >2018-06-24 13:59:38.627 139328 DEBUG cinder.volume.drivers.rbd [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:59:38.681 139328 ERROR cinder.volume.drivers.rbd [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:59:38.681 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:59:38.681 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:59:38.681 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:59:38.681 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:59:38.681 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:59:38.681 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:59:38.684 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:59:38.685 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Have been at this for 0.057 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:59:38.685 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 13:59:38.777 139328 DEBUG cinder.volume.flows.manager.create_volume [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Volume reschedule parameters: True retry: {u'num_attempts': 2, u'backends': [u'hostgroup@tripleo_ceph#tripleo_ceph', u'hostgroup@tripleo_ceph#tripleo_ceph'], u'hosts': [u'hostgroup@tripleo_ceph#tripleo_ceph', u'hostgroup@tripleo_ceph#tripleo_ceph']} get_flow /usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py:1075 >2018-06-24 13:59:38.791 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Flow 'volume_create_manager' (ca46af6c-c063-40df-96fa-ea6fc3b8abd9) transitioned into state 'RUNNING' from state 'PENDING' _flow_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:145 >2018-06-24 13:59:38.793 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (0f9b35ea-0b95-4f85-8333-d134aa140ffb) transitioned into state 'RUNNING' from state 'PENDING' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:194 >2018-06-24 13:59:38.809 139328 DEBUG oslo_db.sqlalchemy.engines [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:285 >2018-06-24 13:59:38.834 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (0f9b35ea-0b95-4f85-8333-d134aa140ffb) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2018-06-24T13:59:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1056713959',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='hostgroup@tripleo_ceph#tripleo_ceph',id=b29f1542-c089-463c-8354-fa3d8de9030e,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='3ee2014cc6684c8484b6a20dcabefd29',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2018-06-24T13:59:38Z,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2018-06-24T13:59:38Z,user_id='239cfa9cffab4bc89f7b738ebd9f59c8',volume_attachment=<?>,volume_type=<?>,volume_type_id=None)' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:183 >2018-06-24 13:59:38.837 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (7450468c-8a0d-4afe-a1cf-dca196c293fe) transitioned into state 'RUNNING' from state 'PENDING' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:194 >2018-06-24 13:59:38.839 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (7450468c-8a0d-4afe-a1cf-dca196c293fe) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:183 >2018-06-24 13:59:38.841 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (4db6ed02-e765-41f1-b15e-78e73759e18d) transitioned into state 'RUNNING' from state 'PENDING' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:194 >2018-06-24 13:59:38.842 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (4db6ed02-e765-41f1-b15e-78e73759e18d) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': u'creating', 'volume_size': 1, 'volume_name': 'volume-b29f1542-c089-463c-8354-fa3d8de9030e', 'type': 'raw', 'volume_id': 'b29f1542-c089-463c-8354-fa3d8de9030e'}' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:183 >2018-06-24 13:59:38.844 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (b6e7db03-c573-46a1-bd67-3ff86330fdeb) transitioned into state 'RUNNING' from state 'PENDING' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:194 >2018-06-24 13:59:38.865 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (b6e7db03-c573-46a1-bd67-3ff86330fdeb) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:183 >2018-06-24 13:59:38.867 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (a75d306b-fa03-4c9e-9c5d-1b7876c7b82c) transitioned into state 'RUNNING' from state 'PENDING' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:194 >2018-06-24 13:59:38.868 139328 INFO cinder.volume.flows.manager.create_volume [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Volume b29f1542-c089-463c-8354-fa3d8de9030e: being created as raw with specification: {'status': u'creating', 'volume_size': 1, 'volume_name': 'volume-b29f1542-c089-463c-8354-fa3d8de9030e'} >2018-06-24 13:59:38.868 139328 DEBUG cinder.volume.drivers.rbd [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] creating volume 'volume-b29f1542-c089-463c-8354-fa3d8de9030e' create_volume /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:655 >2018-06-24 13:59:38.869 139328 DEBUG cinder.volume.drivers.rbd [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:59:38.902 139328 ERROR cinder.volume.drivers.rbd [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:59:38.902 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:59:38.902 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:59:38.902 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:59:38.902 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:59:38.902 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:59:38.902 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:59:38.904 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:59:38.904 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:59:38.905 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 13:59:39.612 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:59:39.612 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 13:59:39.619 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 13:59:39.620 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:59:39.658 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:59:39.658 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:59:39.658 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:59:39.658 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:59:39.658 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:59:39.658 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:59:39.658 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:59:39.660 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:59:39.661 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.041 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:59:39.661 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 13:59:48.686 139328 DEBUG cinder.volume.drivers.rbd [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:59:48.721 139328 ERROR cinder.volume.drivers.rbd [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:59:48.721 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:59:48.721 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:59:48.721 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:59:48.721 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:59:48.721 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:59:48.721 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:59:48.724 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:59:48.724 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Have been at this for 10.096 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:59:48.724 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 13:59:48.905 139328 DEBUG cinder.volume.drivers.rbd [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:59:48.938 139328 ERROR cinder.volume.drivers.rbd [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:59:48.938 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:59:48.938 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:59:48.938 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:59:48.938 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:59:48.938 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:59:48.938 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:59:48.940 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:59:48.941 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:59:48.941 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 13:59:49.662 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 13:59:49.697 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 13:59:49.697 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 13:59:49.697 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 13:59:49.697 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 13:59:49.697 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 13:59:49.697 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 13:59:49.697 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 13:59:49.700 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 13:59:49.700 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.08 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 13:59:49.701 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:00:08.725 139328 DEBUG cinder.volume.drivers.rbd [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:00:08.762 139328 ERROR cinder.volume.drivers.rbd [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:00:08.762 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:00:08.762 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:00:08.762 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:00:08.762 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:00:08.762 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:00:08.762 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:00:08.765 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:00:08.765 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Have been at this for 30.137 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:00:08.767 139328 WARNING cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (8ad5fcce-9c37-43d1-8ae9-6922efda7b9b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' >2018-06-24 14:00:08.769 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (96a893cc-7f92-4fe5-977c-439b6350a172) transitioned into state 'REVERTING' from state 'SUCCESS' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:194 >2018-06-24 14:00:08.771 139328 WARNING cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (96a893cc-7f92-4fe5-977c-439b6350a172) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' >2018-06-24 14:00:08.774 139328 WARNING cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Flow 'volume_create_manager' (147bc965-d641-428a-b91a-8d234430a694) transitioned into state 'REVERTED' from state 'RUNNING' >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Exception during message handling: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server Traceback (most recent call last): >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server File "<string>", line 2, in create_volume >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/objects/cleanable.py", line 207, in wrapper >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 652, in create_volume >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server _run_flow() >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 644, in _run_flow >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server flow_engine.run() >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 247, in run >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 336, in reraise_if_any >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server failures[0].reraise() >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 343, in reraise >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server six.reraise(*self._exc_info) >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server result = task.execute(**arguments) >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 942, in execute >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume(volume, **volume_spec) >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 910, in _create_raw_volume >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 661, in create_volume >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server with RADOSClient(self) as client: >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server return _do_conn(pool, remote, timeout) >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server return r.call(f, *args, **kwargs) >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server raise attempt.get() >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:00:08.775 139328 ERROR oslo_messaging.rpc.server >2018-06-24 14:00:08.941 139328 DEBUG cinder.volume.drivers.rbd [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:00:08.975 139328 ERROR cinder.volume.drivers.rbd [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:00:08.975 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:00:08.975 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:00:08.975 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:00:08.975 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:00:08.975 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:00:08.975 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:00:08.977 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:00:08.977 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:00:08.983 139328 WARNING cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (a75d306b-fa03-4c9e-9c5d-1b7876c7b82c) transitioned into state 'FAILURE' from state 'RUNNING' >5 predecessors (most recent first): > Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'volume': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2018-06-24T13:59:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1056713959',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='hostgroup@tripleo_ceph#tripleo_ceph',id=b29f1542-c089-463c-8354-fa3d8de9030e,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='3ee2014cc6684c8484b6a20dcabefd29',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2018-06-24T13:59:38Z,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2018-06-24T13:59:38Z,user_id='239cfa9cffab4bc89f7b738ebd9f59c8',volume_attachment=<?>,volume_type=<?>,volume_type_id=None), 'context': <cinder.context.RequestContext object at 0x7f796f823910>}, 'provides': None} > |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'volume': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2018-06-24T13:59:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1056713959',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='hostgroup@tripleo_ceph#tripleo_ceph',id=b29f1542-c089-463c-8354-fa3d8de9030e,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='3ee2014cc6684c8484b6a20dcabefd29',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2018-06-24T13:59:38Z,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2018-06-24T13:59:38Z,user_id='239cfa9cffab4bc89f7b738ebd9f59c8',volume_attachment=<?>,volume_type=<?>,volume_type_id=None), 'request_spec': RequestSpec(CG_backend=<?>,cgsnapshot_id=None,consistencygroup_id=None,group_backend=<?>,group_id=None,image_id=None,snapshot_id=None,source_replicaid=None,source_volid=None,volume=Volume(b29f1542-c089-463c-8354-fa3d8de9030e),volume_id=b29f1542-c089-463c-8354-fa3d8de9030e,volume_properties=VolumeProperties,volume_type=None), 'context': <cinder.context.RequestContext object at 0x7f796f823910>}, 'provides': {'status': u'creating', 'volume_size': 1, 'volume_name': 'volume-b29f1542-c089-463c-8354-fa3d8de9030e', 'type': 'raw', 'volume_id': 'b29f1542-c089-463c-8354-fa3d8de9030e'}} > |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'volume': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2018-06-24T13:59:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1056713959',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='hostgroup@tripleo_ceph#tripleo_ceph',id=b29f1542-c089-463c-8354-fa3d8de9030e,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='3ee2014cc6684c8484b6a20dcabefd29',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2018-06-24T13:59:38Z,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2018-06-24T13:59:38Z,user_id='239cfa9cffab4bc89f7b738ebd9f59c8',volume_attachment=<?>,volume_type=<?>,volume_type_id=None), 'filter_properties': {u'config_options': {}, u'request_spec': {u'resource_properties': {u'status': u'creating', u'volume_type_id': None, u'group_id': None, u'user_id': u'239cfa9cffab4bc89f7b738ebd9f59c8', u'availability_zone': u'nova', u'reservations': [u'0e00f6b6-7527-4b5a-9184-b8ac9ffaa4d3', u'b4bce244-a3c7-4874-9e57-3e423c8bc6c6'], u'multiattach': False, u'attach_status': u'detached', u'source_volid': None, u'cgsnapshot_id': None, u'metadata': {}, u'qos_specs': None, u'encryption_key_id': None, u'display_description': None, u'snapshot_id': None, u'source_replicaid': None, u'project_id': u'3ee2014cc6684c8484b6a20dcabefd29', u'consistencygroup_id': None, u'size': 1, u'display_name': u'tempest-TestMinimumBasicScenario-volume-1056713959'}, u'source_replicaid': None, u'volume_properties': {u'status': u'creating', u'volume_type_id': None, u'group_id': None, u'user_id': u'239cfa9cffab4bc89f7b738ebd9f59c8', u'availability_zone': u'nova', u'reservations': [u'0e00f6b6-7527-4b5a-9184-b8ac9ffaa4d3', u'b4bce244-a3c7-4874-9e57-3e423c8bc6c6'], u'multiattach': False, u'attach_status': u'detached', u'source_volid': None, u'cgsnapshot_id': None, u'metadata': {}, u'qos_specs': None, u'encryption_key_id': None, u'display_description': None, u'snapshot_id': None, u'source_replicaid': None, u'project_id': u'3ee2014cc6684c8484b6a20dcabefd29', u'consistencygroup_id': None, u'size': 1, u'display_name': u'tempest-TestMinimumBasicScenario-volume-1056713959'}, u'source_volid': None, u'cgsnapshot_id': None, u'volume': {u'migration_status': None, u'provider_id': None, u'availability_zone': u'nova', u'terminated_at': None, u'updated_at': None, u'provider_geometry': None, u'replication_extended_status': None, u'replication_status': None, u'snapshot_id': None, u'ec2_id': None, u'deleted_at': None, u'id': u'b29f1542-c089-463c-8354-fa3d8de9030e', u'size': 1, u'display_name': u'tempest-TestMinimumBasicScenario-volume-1056713959', u'display_description': None, u'cluster_name': None, u'name_id': u'b29f1542-c089-463c-8354-fa3d8de9030e', u'volume_admin_metadata': [], u'project_id': u'3ee2014cc6684c8484b6a20dcabefd29', u'launched_at': None, u'scheduled_at': None, u'status': u'creating', u'volume_type_id': None, u'multiattach': False, u'deleted': False, u'provider_location': None, u'volume_glance_metadata': [], u'admin_metadata': {}, u'host': None, u'glance_metadata': {}, u'consistencygroup_id': None, u'source_volid': None, u'provider_auth': None, u'previous_status': None, u'group_id': None, u'name': u'volume-b29f1542-c089-463c-8354-fa3d8de9030e', u'user_id': u'239cfa9cffab4bc89f7b738ebd9f59c8', u'bootable': False, u'created_at': u'2018-06-24T13:59:08.000000', u'attach_status': u'detached', u'volume_metadata': [], u'_name_id': None, u'encryption_key_id': None, u'replication_driver_data': None, u'metadata': {}}, u'image_id': None, u'snapshot_id': None, u'consistencygroup_id': None, u'volume_type': None, u'volume_id': u'b29f1542-c089-463c-8354-fa3d8de9030e', u'group_id': None}, u'user_id': u'239cfa9cffab4bc89f7b738ebd9f59c8', u'availability_zone': u'nova', u'volume_type': None, u'qos_specs': None, u'retry': {u'num_attempts': 2, u'backends': [u'hostgroup@tripleo_ceph#tripleo_ceph', u'hostgroup@tripleo_ceph#tripleo_ceph'], u'hosts': [u'hostgroup@tripleo_ceph#tripleo_ceph', u'hostgroup@tripleo_ceph#tripleo_ceph']}, u'metadata': {}, u'resource_type': {}, u'size': 1}, 'context': <cinder.context.RequestContext object at 0x7f796f823910>, 'request_spec': RequestSpec(CG_backend=<?>,cgsnapshot_id=None,consistencygroup_id=None,group_backend=<?>,group_id=None,image_id=None,snapshot_id=None,source_replicaid=None,source_volid=None,volume=Volume(b29f1542-c089-463c-8354-fa3d8de9030e),volume_id=b29f1542-c089-463c-8354-fa3d8de9030e,volume_properties=VolumeProperties,volume_type=None)}, 'provides': None} > |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'volume': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2018-06-24T13:59:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1056713959',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='hostgroup@tripleo_ceph#tripleo_ceph',id=b29f1542-c089-463c-8354-fa3d8de9030e,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='3ee2014cc6684c8484b6a20dcabefd29',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2018-06-24T13:59:38Z,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2018-06-24T13:59:38Z,user_id='239cfa9cffab4bc89f7b738ebd9f59c8',volume_attachment=<?>,volume_type=<?>,volume_type_id=None), 'context': <cinder.context.RequestContext object at 0x7f796f823910>}, 'provides': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2018-06-24T13:59:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1056713959',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='hostgroup@tripleo_ceph#tripleo_ceph',id=b29f1542-c089-463c-8354-fa3d8de9030e,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='3ee2014cc6684c8484b6a20dcabefd29',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2018-06-24T13:59:38Z,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2018-06-24T13:59:38Z,user_id='239cfa9cffab4bc89f7b738ebd9f59c8',volume_attachment=<?>,volume_type=<?>,volume_type_id=None)} > |__Flow 'volume_create_manager': VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:00:08.983 139328 ERROR cinder.volume.manager Traceback (most recent call last): >2018-06-24 14:00:08.983 139328 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task >2018-06-24 14:00:08.983 139328 ERROR cinder.volume.manager result = task.execute(**arguments) >2018-06-24 14:00:08.983 139328 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 942, in execute >2018-06-24 14:00:08.983 139328 ERROR cinder.volume.manager model_update = self._create_raw_volume(volume, **volume_spec) >2018-06-24 14:00:08.983 139328 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 910, in _create_raw_volume >2018-06-24 14:00:08.983 139328 ERROR cinder.volume.manager ret = self.driver.create_volume(volume) >2018-06-24 14:00:08.983 139328 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 661, in create_volume >2018-06-24 14:00:08.983 139328 ERROR cinder.volume.manager with RADOSClient(self) as client: >2018-06-24 14:00:08.983 139328 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:00:08.983 139328 ERROR cinder.volume.manager self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:00:08.983 139328 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:00:08.983 139328 ERROR cinder.volume.manager return _do_conn(pool, remote, timeout) >2018-06-24 14:00:08.983 139328 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:00:08.983 139328 ERROR cinder.volume.manager return r.call(f, *args, **kwargs) >2018-06-24 14:00:08.983 139328 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:00:08.983 139328 ERROR cinder.volume.manager raise attempt.get() >2018-06-24 14:00:08.983 139328 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:00:08.983 139328 ERROR cinder.volume.manager six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:00:08.983 139328 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:00:08.983 139328 ERROR cinder.volume.manager attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:00:08.983 139328 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:00:08.983 139328 ERROR cinder.volume.manager raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:00:08.983 139328 ERROR cinder.volume.manager VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:00:08.983 139328 ERROR cinder.volume.manager >2018-06-24 14:00:08.987 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (a75d306b-fa03-4c9e-9c5d-1b7876c7b82c) transitioned into state 'REVERTING' from state 'FAILURE' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:194 >2018-06-24 14:00:08.989 139328 WARNING cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (a75d306b-fa03-4c9e-9c5d-1b7876c7b82c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' >2018-06-24 14:00:08.991 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (b6e7db03-c573-46a1-bd67-3ff86330fdeb) transitioned into state 'REVERTING' from state 'SUCCESS' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:194 >2018-06-24 14:00:08.992 139328 WARNING cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (b6e7db03-c573-46a1-bd67-3ff86330fdeb) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' >2018-06-24 14:00:08.994 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (4db6ed02-e765-41f1-b15e-78e73759e18d) transitioned into state 'REVERTING' from state 'SUCCESS' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:194 >2018-06-24 14:00:08.996 139328 WARNING cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (4db6ed02-e765-41f1-b15e-78e73759e18d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' >2018-06-24 14:00:08.997 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (7450468c-8a0d-4afe-a1cf-dca196c293fe) transitioned into state 'REVERTING' from state 'SUCCESS' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:194 >2018-06-24 14:00:08.998 139328 DEBUG cinder.volume.flows.manager.create_volume [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Updating volume b29f1542-c089-463c-8354-fa3d8de9030e with {'host': None, 'scheduled_at': datetime.datetime(2018, 6, 24, 14, 0, 8, 998781)}. _pre_reschedule /usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py:121 >2018-06-24 14:00:09.005 139328 DEBUG cinder.volume.flows.manager.create_volume [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Volume b29f1542-c089-463c-8354-fa3d8de9030e: re-scheduling SchedulerAPI.create_volume attempt 2 due to Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. _reschedule /usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py:148 >2018-06-24 14:00:09.022 139328 DEBUG cinder.volume.flows.manager.create_volume [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Volume b29f1542-c089-463c-8354-fa3d8de9030e: re-scheduled _post_reschedule /usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py:160 >2018-06-24 14:00:09.023 139328 DEBUG cinder.volume.drivers.rbd [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:00:09.057 139328 ERROR cinder.volume.drivers.rbd [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:00:09.057 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:00:09.057 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:00:09.057 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:00:09.057 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:00:09.057 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:00:09.057 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:00:09.059 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:00:09.059 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:00:09.060 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:00:09.145 139328 DEBUG cinder.volume.flows.manager.create_volume [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Volume reschedule parameters: True retry: {u'num_attempts': 3, u'backends': [u'hostgroup@tripleo_ceph#tripleo_ceph', u'hostgroup@tripleo_ceph#tripleo_ceph', u'hostgroup@tripleo_ceph#tripleo_ceph'], u'hosts': [u'hostgroup@tripleo_ceph#tripleo_ceph', u'hostgroup@tripleo_ceph#tripleo_ceph', u'hostgroup@tripleo_ceph#tripleo_ceph']} get_flow /usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py:1075 >2018-06-24 14:00:09.159 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Flow 'volume_create_manager' (c8ddde73-604a-49e3-9603-5cbe9702092e) transitioned into state 'RUNNING' from state 'PENDING' _flow_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:145 >2018-06-24 14:00:09.162 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (0ed92503-ed6d-4b7a-91b2-6e05cf32e211) transitioned into state 'RUNNING' from state 'PENDING' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:194 >2018-06-24 14:00:09.181 139328 DEBUG oslo_db.sqlalchemy.engines [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:285 >2018-06-24 14:00:09.202 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (0ed92503-ed6d-4b7a-91b2-6e05cf32e211) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2018-06-24T13:59:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1056713959',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='hostgroup@tripleo_ceph#tripleo_ceph',id=b29f1542-c089-463c-8354-fa3d8de9030e,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='3ee2014cc6684c8484b6a20dcabefd29',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2018-06-24T14:00:09Z,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2018-06-24T14:00:09Z,user_id='239cfa9cffab4bc89f7b738ebd9f59c8',volume_attachment=<?>,volume_type=<?>,volume_type_id=None)' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:183 >2018-06-24 14:00:09.205 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (eba82e4e-4db7-4aa5-9fa9-d399c8b89c71) transitioned into state 'RUNNING' from state 'PENDING' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:194 >2018-06-24 14:00:09.207 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (eba82e4e-4db7-4aa5-9fa9-d399c8b89c71) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:183 >2018-06-24 14:00:09.209 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (934c10b0-c8a7-4820-8733-2f87ca3487a0) transitioned into state 'RUNNING' from state 'PENDING' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:194 >2018-06-24 14:00:09.211 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (934c10b0-c8a7-4820-8733-2f87ca3487a0) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': u'creating', 'volume_size': 1, 'volume_name': 'volume-b29f1542-c089-463c-8354-fa3d8de9030e', 'type': 'raw', 'volume_id': 'b29f1542-c089-463c-8354-fa3d8de9030e'}' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:183 >2018-06-24 14:00:09.213 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (ba4a7b93-b535-4b8c-a0a5-ef3517cb79a0) transitioned into state 'RUNNING' from state 'PENDING' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:194 >2018-06-24 14:00:09.241 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (ba4a7b93-b535-4b8c-a0a5-ef3517cb79a0) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:183 >2018-06-24 14:00:09.243 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (60c5f342-99e6-4ea2-9e9e-b547cf1c25c6) transitioned into state 'RUNNING' from state 'PENDING' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:194 >2018-06-24 14:00:09.243 139328 INFO cinder.volume.flows.manager.create_volume [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Volume b29f1542-c089-463c-8354-fa3d8de9030e: being created as raw with specification: {'status': u'creating', 'volume_size': 1, 'volume_name': 'volume-b29f1542-c089-463c-8354-fa3d8de9030e'} >2018-06-24 14:00:09.244 139328 DEBUG cinder.volume.drivers.rbd [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] creating volume 'volume-b29f1542-c089-463c-8354-fa3d8de9030e' create_volume /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:655 >2018-06-24 14:00:09.244 139328 DEBUG cinder.volume.drivers.rbd [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:00:09.280 139328 ERROR cinder.volume.drivers.rbd [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:00:09.280 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:00:09.280 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:00:09.280 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:00:09.280 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:00:09.280 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:00:09.280 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:00:09.282 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:00:09.282 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:00:09.283 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:00:09.702 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:00:09.736 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:00:09.736 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:00:09.736 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:00:09.736 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:00:09.736 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:00:09.736 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:00:09.736 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:00:09.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:00:09.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.12 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:00:09.740 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:00:09.740 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:00:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:00:09.740 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:00:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:00:09.740 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:00:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:00:09.740 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:00:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:00:09.740 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:00:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:00:09.740 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:00:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:00:09.740 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:00:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:00:09.740 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:00:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:00:09.740 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:00:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:00:09.740 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:00:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:00:09.740 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:00:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:00:09.740 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:00:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:00:09.740 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:00:09.740 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:00:09.740 139328 ERROR oslo_service.periodic_task >2018-06-24 14:00:19.060 139328 DEBUG cinder.volume.drivers.rbd [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:00:19.098 139328 ERROR cinder.volume.drivers.rbd [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:00:19.098 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:00:19.098 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:00:19.098 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:00:19.098 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:00:19.098 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:00:19.098 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:00:19.100 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:00:19.100 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:00:19.101 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:00:19.283 139328 DEBUG cinder.volume.drivers.rbd [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:00:19.318 139328 ERROR cinder.volume.drivers.rbd [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:00:19.318 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:00:19.318 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:00:19.318 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:00:19.318 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:00:19.318 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:00:19.318 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:00:19.321 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:00:19.321 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:00:19.321 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:00:39.101 139328 DEBUG cinder.volume.drivers.rbd [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:00:39.137 139328 ERROR cinder.volume.drivers.rbd [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:00:39.137 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:00:39.137 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:00:39.137 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:00:39.137 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:00:39.137 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:00:39.137 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:00:39.139 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:00:39.140 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:00:39.141 139328 WARNING cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (7450468c-8a0d-4afe-a1cf-dca196c293fe) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' >2018-06-24 14:00:39.143 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (0f9b35ea-0b95-4f85-8333-d134aa140ffb) transitioned into state 'REVERTING' from state 'SUCCESS' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:194 >2018-06-24 14:00:39.145 139328 WARNING cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (0f9b35ea-0b95-4f85-8333-d134aa140ffb) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' >2018-06-24 14:00:39.147 139328 WARNING cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Flow 'volume_create_manager' (ca46af6c-c063-40df-96fa-ea6fc3b8abd9) transitioned into state 'REVERTED' from state 'RUNNING' >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Exception during message handling: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server Traceback (most recent call last): >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server File "<string>", line 2, in create_volume >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/objects/cleanable.py", line 207, in wrapper >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 652, in create_volume >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server _run_flow() >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 644, in _run_flow >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server flow_engine.run() >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 247, in run >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 336, in reraise_if_any >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server failures[0].reraise() >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 343, in reraise >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server six.reraise(*self._exc_info) >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server result = task.execute(**arguments) >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 942, in execute >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume(volume, **volume_spec) >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 910, in _create_raw_volume >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 661, in create_volume >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server with RADOSClient(self) as client: >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server return _do_conn(pool, remote, timeout) >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server return r.call(f, *args, **kwargs) >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server raise attempt.get() >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:00:39.148 139328 ERROR oslo_messaging.rpc.server >2018-06-24 14:00:39.322 139328 DEBUG cinder.volume.drivers.rbd [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:00:39.354 139328 ERROR cinder.volume.drivers.rbd [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:00:39.354 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:00:39.354 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:00:39.354 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:00:39.354 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:00:39.354 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:00:39.354 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:00:39.357 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:00:39.357 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:00:39.363 139328 WARNING cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (60c5f342-99e6-4ea2-9e9e-b547cf1c25c6) transitioned into state 'FAILURE' from state 'RUNNING' >5 predecessors (most recent first): > Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'volume': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2018-06-24T13:59:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1056713959',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='hostgroup@tripleo_ceph#tripleo_ceph',id=b29f1542-c089-463c-8354-fa3d8de9030e,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='3ee2014cc6684c8484b6a20dcabefd29',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2018-06-24T14:00:09Z,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2018-06-24T14:00:09Z,user_id='239cfa9cffab4bc89f7b738ebd9f59c8',volume_attachment=<?>,volume_type=<?>,volume_type_id=None), 'context': <cinder.context.RequestContext object at 0x7f796fafed10>}, 'provides': None} > |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'volume': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2018-06-24T13:59:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1056713959',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='hostgroup@tripleo_ceph#tripleo_ceph',id=b29f1542-c089-463c-8354-fa3d8de9030e,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='3ee2014cc6684c8484b6a20dcabefd29',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2018-06-24T14:00:09Z,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2018-06-24T14:00:09Z,user_id='239cfa9cffab4bc89f7b738ebd9f59c8',volume_attachment=<?>,volume_type=<?>,volume_type_id=None), 'request_spec': RequestSpec(CG_backend=<?>,cgsnapshot_id=None,consistencygroup_id=None,group_backend=<?>,group_id=None,image_id=None,snapshot_id=None,source_replicaid=None,source_volid=None,volume=Volume(b29f1542-c089-463c-8354-fa3d8de9030e),volume_id=b29f1542-c089-463c-8354-fa3d8de9030e,volume_properties=VolumeProperties,volume_type=None), 'context': <cinder.context.RequestContext object at 0x7f796fafed10>}, 'provides': {'status': u'creating', 'volume_size': 1, 'volume_name': 'volume-b29f1542-c089-463c-8354-fa3d8de9030e', 'type': 'raw', 'volume_id': 'b29f1542-c089-463c-8354-fa3d8de9030e'}} > |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'volume': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2018-06-24T13:59:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1056713959',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='hostgroup@tripleo_ceph#tripleo_ceph',id=b29f1542-c089-463c-8354-fa3d8de9030e,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='3ee2014cc6684c8484b6a20dcabefd29',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2018-06-24T14:00:09Z,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2018-06-24T14:00:09Z,user_id='239cfa9cffab4bc89f7b738ebd9f59c8',volume_attachment=<?>,volume_type=<?>,volume_type_id=None), 'filter_properties': {u'config_options': {}, u'request_spec': {u'resource_properties': {u'status': u'creating', u'volume_type_id': None, u'group_id': None, u'user_id': u'239cfa9cffab4bc89f7b738ebd9f59c8', u'availability_zone': u'nova', u'reservations': [u'0e00f6b6-7527-4b5a-9184-b8ac9ffaa4d3', u'b4bce244-a3c7-4874-9e57-3e423c8bc6c6'], u'multiattach': False, u'attach_status': u'detached', u'source_volid': None, u'cgsnapshot_id': None, u'metadata': {}, u'qos_specs': None, u'encryption_key_id': None, u'display_description': None, u'snapshot_id': None, u'source_replicaid': None, u'project_id': u'3ee2014cc6684c8484b6a20dcabefd29', u'consistencygroup_id': None, u'size': 1, u'display_name': u'tempest-TestMinimumBasicScenario-volume-1056713959'}, u'source_replicaid': None, u'volume_properties': {u'status': u'creating', u'volume_type_id': None, u'group_id': None, u'user_id': u'239cfa9cffab4bc89f7b738ebd9f59c8', u'availability_zone': u'nova', u'reservations': [u'0e00f6b6-7527-4b5a-9184-b8ac9ffaa4d3', u'b4bce244-a3c7-4874-9e57-3e423c8bc6c6'], u'multiattach': False, u'attach_status': u'detached', u'source_volid': None, u'cgsnapshot_id': None, u'metadata': {}, u'qos_specs': None, u'encryption_key_id': None, u'display_description': None, u'snapshot_id': None, u'source_replicaid': None, u'project_id': u'3ee2014cc6684c8484b6a20dcabefd29', u'consistencygroup_id': None, u'size': 1, u'display_name': u'tempest-TestMinimumBasicScenario-volume-1056713959'}, u'source_volid': None, u'cgsnapshot_id': None, u'volume': {u'migration_status': None, u'provider_id': None, u'availability_zone': u'nova', u'terminated_at': None, u'updated_at': None, u'provider_geometry': None, u'replication_extended_status': None, u'replication_status': None, u'snapshot_id': None, u'ec2_id': None, u'deleted_at': None, u'id': u'b29f1542-c089-463c-8354-fa3d8de9030e', u'size': 1, u'display_name': u'tempest-TestMinimumBasicScenario-volume-1056713959', u'display_description': None, u'cluster_name': None, u'name_id': u'b29f1542-c089-463c-8354-fa3d8de9030e', u'volume_admin_metadata': [], u'project_id': u'3ee2014cc6684c8484b6a20dcabefd29', u'launched_at': None, u'scheduled_at': None, u'status': u'creating', u'volume_type_id': None, u'multiattach': False, u'deleted': False, u'provider_location': None, u'volume_glance_metadata': [], u'admin_metadata': {}, u'host': None, u'glance_metadata': {}, u'consistencygroup_id': None, u'source_volid': None, u'provider_auth': None, u'previous_status': None, u'group_id': None, u'name': u'volume-b29f1542-c089-463c-8354-fa3d8de9030e', u'user_id': u'239cfa9cffab4bc89f7b738ebd9f59c8', u'bootable': False, u'created_at': u'2018-06-24T13:59:08.000000', u'attach_status': u'detached', u'volume_metadata': [], u'_name_id': None, u'encryption_key_id': None, u'replication_driver_data': None, u'metadata': {}}, u'image_id': None, u'snapshot_id': None, u'consistencygroup_id': None, u'volume_type': None, u'volume_id': u'b29f1542-c089-463c-8354-fa3d8de9030e', u'group_id': None}, u'user_id': u'239cfa9cffab4bc89f7b738ebd9f59c8', u'availability_zone': u'nova', u'volume_type': None, u'qos_specs': None, u'retry': {u'num_attempts': 3, u'backends': [u'hostgroup@tripleo_ceph#tripleo_ceph', u'hostgroup@tripleo_ceph#tripleo_ceph', u'hostgroup@tripleo_ceph#tripleo_ceph'], u'hosts': [u'hostgroup@tripleo_ceph#tripleo_ceph', u'hostgroup@tripleo_ceph#tripleo_ceph', u'hostgroup@tripleo_ceph#tripleo_ceph']}, u'metadata': {}, u'resource_type': {}, u'size': 1}, 'context': <cinder.context.RequestContext object at 0x7f796fafed10>, 'request_spec': RequestSpec(CG_backend=<?>,cgsnapshot_id=None,consistencygroup_id=None,group_backend=<?>,group_id=None,image_id=None,snapshot_id=None,source_replicaid=None,source_volid=None,volume=Volume(b29f1542-c089-463c-8354-fa3d8de9030e),volume_id=b29f1542-c089-463c-8354-fa3d8de9030e,volume_properties=VolumeProperties,volume_type=None)}, 'provides': None} > |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'volume': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2018-06-24T13:59:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1056713959',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='hostgroup@tripleo_ceph#tripleo_ceph',id=b29f1542-c089-463c-8354-fa3d8de9030e,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='3ee2014cc6684c8484b6a20dcabefd29',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2018-06-24T14:00:09Z,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2018-06-24T14:00:09Z,user_id='239cfa9cffab4bc89f7b738ebd9f59c8',volume_attachment=<?>,volume_type=<?>,volume_type_id=None), 'context': <cinder.context.RequestContext object at 0x7f796fafed10>}, 'provides': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2018-06-24T13:59:08Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TestMinimumBasicScenario-volume-1056713959',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='hostgroup@tripleo_ceph#tripleo_ceph',id=b29f1542-c089-463c-8354-fa3d8de9030e,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='3ee2014cc6684c8484b6a20dcabefd29',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2018-06-24T14:00:09Z,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2018-06-24T14:00:09Z,user_id='239cfa9cffab4bc89f7b738ebd9f59c8',volume_attachment=<?>,volume_type=<?>,volume_type_id=None)} > |__Flow 'volume_create_manager': VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:00:39.363 139328 ERROR cinder.volume.manager Traceback (most recent call last): >2018-06-24 14:00:39.363 139328 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task >2018-06-24 14:00:39.363 139328 ERROR cinder.volume.manager result = task.execute(**arguments) >2018-06-24 14:00:39.363 139328 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 942, in execute >2018-06-24 14:00:39.363 139328 ERROR cinder.volume.manager model_update = self._create_raw_volume(volume, **volume_spec) >2018-06-24 14:00:39.363 139328 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 910, in _create_raw_volume >2018-06-24 14:00:39.363 139328 ERROR cinder.volume.manager ret = self.driver.create_volume(volume) >2018-06-24 14:00:39.363 139328 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 661, in create_volume >2018-06-24 14:00:39.363 139328 ERROR cinder.volume.manager with RADOSClient(self) as client: >2018-06-24 14:00:39.363 139328 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:00:39.363 139328 ERROR cinder.volume.manager self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:00:39.363 139328 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:00:39.363 139328 ERROR cinder.volume.manager return _do_conn(pool, remote, timeout) >2018-06-24 14:00:39.363 139328 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:00:39.363 139328 ERROR cinder.volume.manager return r.call(f, *args, **kwargs) >2018-06-24 14:00:39.363 139328 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:00:39.363 139328 ERROR cinder.volume.manager raise attempt.get() >2018-06-24 14:00:39.363 139328 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:00:39.363 139328 ERROR cinder.volume.manager six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:00:39.363 139328 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:00:39.363 139328 ERROR cinder.volume.manager attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:00:39.363 139328 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:00:39.363 139328 ERROR cinder.volume.manager raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:00:39.363 139328 ERROR cinder.volume.manager VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:00:39.363 139328 ERROR cinder.volume.manager >2018-06-24 14:00:39.367 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (60c5f342-99e6-4ea2-9e9e-b547cf1c25c6) transitioned into state 'REVERTING' from state 'FAILURE' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:194 >2018-06-24 14:00:39.369 139328 WARNING cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (60c5f342-99e6-4ea2-9e9e-b547cf1c25c6) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' >2018-06-24 14:00:39.371 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (ba4a7b93-b535-4b8c-a0a5-ef3517cb79a0) transitioned into state 'REVERTING' from state 'SUCCESS' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:194 >2018-06-24 14:00:39.373 139328 WARNING cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (ba4a7b93-b535-4b8c-a0a5-ef3517cb79a0) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' >2018-06-24 14:00:39.375 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (934c10b0-c8a7-4820-8733-2f87ca3487a0) transitioned into state 'REVERTING' from state 'SUCCESS' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:194 >2018-06-24 14:00:39.377 139328 WARNING cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (934c10b0-c8a7-4820-8733-2f87ca3487a0) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' >2018-06-24 14:00:39.379 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (eba82e4e-4db7-4aa5-9fa9-d399c8b89c71) transitioned into state 'REVERTING' from state 'SUCCESS' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:194 >2018-06-24 14:00:39.380 139328 DEBUG cinder.volume.flows.manager.create_volume [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Updating volume b29f1542-c089-463c-8354-fa3d8de9030e with {'host': None, 'scheduled_at': datetime.datetime(2018, 6, 24, 14, 0, 39, 380211)}. _pre_reschedule /usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py:121 >2018-06-24 14:00:39.387 139328 DEBUG cinder.volume.flows.manager.create_volume [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Volume b29f1542-c089-463c-8354-fa3d8de9030e: re-scheduling SchedulerAPI.create_volume attempt 3 due to Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. _reschedule /usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py:148 >2018-06-24 14:00:39.403 139328 DEBUG cinder.volume.flows.manager.create_volume [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Volume b29f1542-c089-463c-8354-fa3d8de9030e: re-scheduled _post_reschedule /usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py:160 >2018-06-24 14:00:39.404 139328 DEBUG cinder.volume.drivers.rbd [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:00:39.436 139328 ERROR cinder.volume.drivers.rbd [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:00:39.436 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:00:39.436 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:00:39.436 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:00:39.436 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:00:39.436 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:00:39.436 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:00:39.438 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:00:39.439 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:00:39.439 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:00:39.611 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:00:39.612 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:00:39.619 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:00:39.620 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:00:39.668 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:00:39.668 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:00:39.668 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:00:39.668 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:00:39.668 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:00:39.668 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:00:39.668 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:00:39.671 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:00:39.671 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.05 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:00:39.672 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:00:49.440 139328 DEBUG cinder.volume.drivers.rbd [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:00:49.478 139328 ERROR cinder.volume.drivers.rbd [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:00:49.478 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:00:49.478 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:00:49.478 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:00:49.478 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:00:49.478 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:00:49.478 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:00:49.481 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:00:49.481 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:00:49.482 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:00:49.672 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:00:49.710 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:00:49.710 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:00:49.710 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:00:49.710 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:00:49.710 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:00:49.710 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:00:49.710 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:00:49.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:00:49.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.092 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:00:49.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:01:09.482 139328 DEBUG cinder.volume.drivers.rbd [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:01:09.522 139328 ERROR cinder.volume.drivers.rbd [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:01:09.522 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:01:09.522 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:01:09.522 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:01:09.522 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:01:09.522 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:01:09.522 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:01:09.524 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:01:09.525 139328 DEBUG cinder.utils [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Have been at this for 30.121 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:01:09.527 139328 WARNING cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (eba82e4e-4db7-4aa5-9fa9-d399c8b89c71) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True' >2018-06-24 14:01:09.528 139328 DEBUG cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (0ed92503-ed6d-4b7a-91b2-6e05cf32e211) transitioned into state 'REVERTING' from state 'SUCCESS' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:194 >2018-06-24 14:01:09.530 139328 WARNING cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (0ed92503-ed6d-4b7a-91b2-6e05cf32e211) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' >2018-06-24 14:01:09.532 139328 WARNING cinder.volume.manager [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Flow 'volume_create_manager' (c8ddde73-604a-49e3-9603-5cbe9702092e) transitioned into state 'REVERTED' from state 'RUNNING' >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server [req-63f464eb-1c45-47c5-bb28-65961e13c809 239cfa9cffab4bc89f7b738ebd9f59c8 3ee2014cc6684c8484b6a20dcabefd29 - default default] Exception during message handling: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server Traceback (most recent call last): >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server File "<string>", line 2, in create_volume >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/objects/cleanable.py", line 207, in wrapper >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 652, in create_volume >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server _run_flow() >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 644, in _run_flow >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server flow_engine.run() >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 247, in run >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 336, in reraise_if_any >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server failures[0].reraise() >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 343, in reraise >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server six.reraise(*self._exc_info) >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server result = task.execute(**arguments) >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 942, in execute >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume(volume, **volume_spec) >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 910, in _create_raw_volume >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume) >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 661, in create_volume >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server with RADOSClient(self) as client: >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server return _do_conn(pool, remote, timeout) >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server return r.call(f, *args, **kwargs) >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server raise attempt.get() >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:01:09.533 139328 ERROR oslo_messaging.rpc.server >2018-06-24 14:01:09.714 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:01:09.747 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:01:09.747 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:01:09.747 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:01:09.747 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:01:09.747 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:01:09.747 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:01:09.747 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:01:09.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:01:09.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.129 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:01:09.750 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:01:09.750 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:01:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:01:09.750 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:01:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:01:09.750 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:01:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:01:09.750 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:01:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:01:09.750 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:01:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:01:09.750 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:01:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:01:09.750 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:01:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:01:09.750 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:01:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:01:09.750 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:01:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:01:09.750 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:01:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:01:09.750 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:01:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:01:09.750 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:01:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:01:09.750 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:01:09.750 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:01:09.750 139328 ERROR oslo_service.periodic_task >2018-06-24 14:01:39.611 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:01:39.614 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:01:39.626 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:01:39.627 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:01:39.666 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:01:39.666 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:01:39.666 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:01:39.666 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:01:39.666 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:01:39.666 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:01:39.666 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:01:39.669 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:01:39.669 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.041 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:01:39.669 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:01:49.670 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:01:49.713 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:01:49.713 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:01:49.713 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:01:49.713 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:01:49.713 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:01:49.713 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:01:49.713 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:01:49.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:01:49.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.088 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:01:49.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:02:09.718 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:02:09.756 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:02:09.756 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:02:09.756 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:02:09.756 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:02:09.756 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:02:09.756 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:02:09.756 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:02:09.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:02:09.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.131 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:02:09.759 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:02:09.759 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:02:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:02:09.759 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:02:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:02:09.759 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:02:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:02:09.759 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:02:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:02:09.759 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:02:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:02:09.759 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:02:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:02:09.759 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:02:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:02:09.759 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:02:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:02:09.759 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:02:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:02:09.759 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:02:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:02:09.759 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:02:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:02:09.759 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:02:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:02:09.759 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:02:09.759 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:02:09.759 139328 ERROR oslo_service.periodic_task >2018-06-24 14:02:39.610 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:02:39.611 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:02:39.622 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:02:39.623 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:02:39.657 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:02:39.657 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:02:39.657 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:02:39.657 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:02:39.657 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:02:39.657 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:02:39.657 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:02:39.658 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:02:39.659 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:02:39.659 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:02:49.660 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:02:49.699 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:02:49.699 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:02:49.699 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:02:49.699 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:02:49.699 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:02:49.699 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:02:49.699 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:02:49.701 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:02:49.702 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:02:49.702 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:03:09.703 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:03:09.742 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:03:09.742 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:03:09.742 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:03:09.742 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:03:09.742 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:03:09.742 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:03:09.742 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:03:09.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:03:09.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.123 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:03:09.747 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:03:09.747 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:03:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:03:09.747 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:03:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:03:09.747 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:03:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:03:09.747 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:03:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:03:09.747 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:03:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:03:09.747 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:03:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:03:09.747 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:03:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:03:09.747 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:03:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:03:09.747 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:03:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:03:09.747 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:03:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:03:09.747 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:03:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:03:09.747 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:03:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:03:09.747 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:03:09.747 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:03:09.747 139328 ERROR oslo_service.periodic_task >2018-06-24 14:03:39.609 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:03:39.610 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:03:39.624 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:03:39.625 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:03:39.670 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:03:39.670 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:03:39.670 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:03:39.670 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:03:39.670 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:03:39.670 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:03:39.670 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:03:39.672 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:03:39.672 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.047 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:03:39.673 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:03:49.673 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:03:49.707 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:03:49.707 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:03:49.707 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:03:49.707 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:03:49.707 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:03:49.707 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:03:49.707 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:03:49.710 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:03:49.710 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.086 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:03:49.711 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:04:09.711 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:04:09.748 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:04:09.748 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:04:09.748 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:04:09.748 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:04:09.748 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:04:09.748 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:04:09.748 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:04:09.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:04:09.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.126 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:04:09.751 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:04:09.751 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:04:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:04:09.751 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:04:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:04:09.751 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:04:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:04:09.751 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:04:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:04:09.751 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:04:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:04:09.751 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:04:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:04:09.751 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:04:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:04:09.751 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:04:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:04:09.751 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:04:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:04:09.751 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:04:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:04:09.751 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:04:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:04:09.751 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:04:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:04:09.751 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:04:09.751 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:04:09.751 139328 ERROR oslo_service.periodic_task >2018-06-24 14:04:39.613 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:04:39.615 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:04:39.626 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:04:39.627 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:04:39.666 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:04:39.666 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:04:39.666 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:04:39.666 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:04:39.666 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:04:39.666 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:04:39.666 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:04:39.668 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:04:39.669 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.042 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:04:39.669 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:04:49.670 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:04:49.706 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:04:49.706 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:04:49.706 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:04:49.706 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:04:49.706 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:04:49.706 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:04:49.706 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:04:49.708 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:04:49.708 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.082 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:04:49.709 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:05:09.709 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:05:09.744 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:05:09.744 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:05:09.744 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:05:09.744 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:05:09.744 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:05:09.744 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:05:09.744 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:05:09.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:05:09.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.12 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:05:09.748 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:05:09.748 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:05:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:05:09.748 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:05:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:05:09.748 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:05:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:05:09.748 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:05:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:05:09.748 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:05:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:05:09.748 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:05:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:05:09.748 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:05:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:05:09.748 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:05:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:05:09.748 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:05:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:05:09.748 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:05:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:05:09.748 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:05:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:05:09.748 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:05:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:05:09.748 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:05:09.748 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:05:09.748 139328 ERROR oslo_service.periodic_task >2018-06-24 14:05:39.609 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:05:39.610 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:05:39.617 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:05:39.617 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:05:39.653 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:05:39.653 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:05:39.653 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:05:39.653 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:05:39.653 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:05:39.653 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:05:39.653 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:05:39.656 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:05:39.656 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:05:39.656 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:05:49.657 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:05:49.689 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:05:49.689 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:05:49.689 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:05:49.689 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:05:49.689 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:05:49.689 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:05:49.689 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:05:49.692 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:05:49.692 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:05:49.692 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:06:09.693 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:06:09.728 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:06:09.728 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:06:09.728 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:06:09.728 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:06:09.728 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:06:09.728 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:06:09.728 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:06:09.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:06:09.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:06:09.730 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:06:09.730 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:06:09.730 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:06:09.730 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:06:09.730 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:06:09.730 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:06:09.730 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:06:09.730 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:06:09.730 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:06:09.730 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:06:09.730 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:06:09.730 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:06:09.730 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:06:09.730 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:06:09.730 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:06:09.730 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:06:09.730 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:06:09.730 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:06:09.730 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:06:09.730 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:06:09.730 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:06:09.730 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:06:09.730 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:06:09.730 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:06:09.730 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:06:09.730 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:06:09.730 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:06:09.730 139328 ERROR oslo_service.periodic_task >2018-06-24 14:06:39.612 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:06:39.613 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:06:39.649 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:06:39.649 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:06:39.649 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:06:39.649 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:06:39.649 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:06:39.649 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:06:39.649 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:06:39.652 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:06:39.652 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:06:39.653 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:06:49.653 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:06:49.687 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:06:49.687 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:06:49.687 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:06:49.687 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:06:49.687 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:06:49.687 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:06:49.687 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:06:49.690 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:06:49.691 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:06:49.691 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:07:09.692 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:07:09.733 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:07:09.733 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:07:09.733 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:07:09.733 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:07:09.733 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:07:09.733 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:07:09.733 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:07:09.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:07:09.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.121 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:07:09.735 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:07:09.735 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:07:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:07:09.735 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:07:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:07:09.735 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:07:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:07:09.735 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:07:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:07:09.735 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:07:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:07:09.735 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:07:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:07:09.735 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:07:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:07:09.735 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:07:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:07:09.735 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:07:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:07:09.735 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:07:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:07:09.735 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:07:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:07:09.735 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:07:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:07:09.735 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:07:09.735 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:07:09.735 139328 ERROR oslo_service.periodic_task >2018-06-24 14:07:39.617 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:07:39.617 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:07:39.628 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:07:39.629 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:07:39.665 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:07:39.665 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:07:39.665 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:07:39.665 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:07:39.665 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:07:39.665 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:07:39.665 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:07:39.667 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:07:39.668 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:07:39.668 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:07:49.669 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:07:49.703 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:07:49.703 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:07:49.703 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:07:49.703 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:07:49.703 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:07:49.703 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:07:49.703 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:07:49.705 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:07:49.705 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:07:49.706 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:08:09.706 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:08:09.742 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:08:09.742 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:08:09.742 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:08:09.742 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:08:09.742 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:08:09.742 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:08:09.742 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:08:09.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:08:09.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:08:09.746 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:08:09.746 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:08:09.746 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:08:09.746 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:08:09.746 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:08:09.746 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:08:09.746 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:08:09.746 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:08:09.746 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:08:09.746 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:08:09.746 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:08:09.746 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:08:09.746 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:08:09.746 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:08:09.746 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:08:09.746 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:08:09.746 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:08:09.746 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:08:09.746 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:08:09.746 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:08:09.746 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:08:09.746 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:08:09.746 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:08:09.746 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:08:09.746 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:08:09.746 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:08:09.746 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:08:09.746 139328 ERROR oslo_service.periodic_task >2018-06-24 14:08:39.617 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:08:39.618 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:08:39.625 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:08:39.626 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:08:39.662 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:08:39.662 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:08:39.662 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:08:39.662 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:08:39.662 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:08:39.662 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:08:39.662 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:08:39.664 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:08:39.664 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:08:39.665 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:08:49.665 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:08:49.697 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:08:49.697 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:08:49.697 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:08:49.697 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:08:49.697 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:08:49.697 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:08:49.697 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:08:49.699 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:08:49.700 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:08:49.700 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:09:09.701 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:09:09.735 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:09:09.735 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:09:09.735 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:09:09.735 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:09:09.735 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:09:09.735 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:09:09.735 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:09:09.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:09:09.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:09:09.738 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:09:09.738 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:09:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:09:09.738 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:09:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:09:09.738 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:09:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:09:09.738 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:09:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:09:09.738 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:09:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:09:09.738 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:09:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:09:09.738 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:09:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:09:09.738 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:09:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:09:09.738 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:09:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:09:09.738 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:09:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:09:09.738 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:09:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:09:09.738 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:09:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:09:09.738 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:09:09.738 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:09:09.738 139328 ERROR oslo_service.periodic_task >2018-06-24 14:09:39.620 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:09:39.620 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:09:39.628 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:09:39.628 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:09:39.664 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:09:39.664 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:09:39.664 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:09:39.664 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:09:39.664 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:09:39.664 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:09:39.664 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:09:39.667 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:09:39.667 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:09:39.667 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:09:49.668 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:09:49.702 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:09:49.702 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:09:49.702 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:09:49.702 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:09:49.702 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:09:49.702 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:09:49.702 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:09:49.704 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:09:49.704 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:09:49.705 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:10:09.705 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:10:09.741 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:10:09.741 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:10:09.741 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:10:09.741 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:10:09.741 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:10:09.741 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:10:09.741 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:10:09.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:10:09.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:10:09.744 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:10:09.744 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:10:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:10:09.744 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:10:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:10:09.744 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:10:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:10:09.744 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:10:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:10:09.744 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:10:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:10:09.744 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:10:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:10:09.744 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:10:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:10:09.744 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:10:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:10:09.744 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:10:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:10:09.744 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:10:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:10:09.744 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:10:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:10:09.744 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:10:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:10:09.744 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:10:09.744 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:10:09.744 139328 ERROR oslo_service.periodic_task >2018-06-24 14:10:39.615 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:10:39.616 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:10:39.623 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:10:39.624 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:10:39.658 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:10:39.658 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:10:39.658 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:10:39.658 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:10:39.658 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:10:39.658 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:10:39.658 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:10:39.660 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:10:39.660 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:10:39.661 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:10:49.661 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:10:49.705 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:10:49.705 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:10:49.705 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:10:49.705 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:10:49.705 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:10:49.705 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:10:49.705 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:10:49.707 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:10:49.708 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.084 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:10:49.708 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:11:09.708 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:11:09.743 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:11:09.743 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:11:09.743 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:11:09.743 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:11:09.743 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:11:09.743 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:11:09.743 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:11:09.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:11:09.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.122 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:11:09.747 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:11:09.747 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:11:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:11:09.747 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:11:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:11:09.747 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:11:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:11:09.747 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:11:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:11:09.747 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:11:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:11:09.747 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:11:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:11:09.747 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:11:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:11:09.747 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:11:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:11:09.747 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:11:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:11:09.747 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:11:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:11:09.747 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:11:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:11:09.747 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:11:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:11:09.747 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:11:09.747 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:11:09.747 139328 ERROR oslo_service.periodic_task >2018-06-24 14:11:39.618 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:11:39.619 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:11:39.627 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:11:39.627 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:11:39.663 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:11:39.663 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:11:39.663 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:11:39.663 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:11:39.663 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:11:39.663 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:11:39.663 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:11:39.665 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:11:39.665 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:11:39.666 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:11:49.666 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:11:49.699 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:11:49.699 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:11:49.699 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:11:49.699 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:11:49.699 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:11:49.699 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:11:49.699 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:11:49.703 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:11:49.704 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:11:49.704 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:12:09.705 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:12:09.741 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:12:09.741 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:12:09.741 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:12:09.741 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:12:09.741 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:12:09.741 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:12:09.741 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:12:09.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:12:09.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:12:09.744 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:12:09.744 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:12:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:12:09.744 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:12:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:12:09.744 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:12:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:12:09.744 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:12:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:12:09.744 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:12:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:12:09.744 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:12:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:12:09.744 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:12:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:12:09.744 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:12:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:12:09.744 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:12:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:12:09.744 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:12:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:12:09.744 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:12:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:12:09.744 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:12:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:12:09.744 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:12:09.744 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:12:09.744 139328 ERROR oslo_service.periodic_task >2018-06-24 14:12:39.615 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:12:39.616 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:12:39.622 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:12:39.623 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:12:39.657 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:12:39.657 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:12:39.657 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:12:39.657 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:12:39.657 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:12:39.657 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:12:39.657 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:12:39.660 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:12:39.660 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:12:39.660 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:12:49.661 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:12:49.694 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:12:49.694 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:12:49.694 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:12:49.694 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:12:49.694 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:12:49.694 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:12:49.694 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:12:49.698 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:12:49.698 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:12:49.699 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:13:09.699 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:13:09.733 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:13:09.733 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:13:09.733 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:13:09.733 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:13:09.733 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:13:09.733 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:13:09.733 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:13:09.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:13:09.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:13:09.737 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:13:09.737 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:13:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:13:09.737 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:13:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:13:09.737 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:13:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:13:09.737 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:13:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:13:09.737 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:13:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:13:09.737 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:13:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:13:09.737 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:13:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:13:09.737 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:13:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:13:09.737 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:13:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:13:09.737 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:13:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:13:09.737 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:13:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:13:09.737 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:13:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:13:09.737 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:13:09.737 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:13:09.737 139328 ERROR oslo_service.periodic_task >2018-06-24 14:13:39.619 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:13:39.620 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:13:39.627 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:13:39.628 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:13:39.665 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:13:39.665 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:13:39.665 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:13:39.665 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:13:39.665 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:13:39.665 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:13:39.665 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:13:39.668 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:13:39.668 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:13:39.668 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:13:49.670 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:13:49.703 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:13:49.703 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:13:49.703 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:13:49.703 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:13:49.703 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:13:49.703 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:13:49.703 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:13:49.706 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:13:49.706 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:13:49.707 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:14:09.707 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:14:09.744 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:14:09.744 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:14:09.744 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:14:09.744 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:14:09.744 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:14:09.744 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:14:09.744 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:14:09.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:14:09.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.119 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:14:09.747 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:14:09.747 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:14:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:14:09.747 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:14:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:14:09.747 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:14:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:14:09.747 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:14:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:14:09.747 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:14:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:14:09.747 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:14:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:14:09.747 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:14:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:14:09.747 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:14:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:14:09.747 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:14:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:14:09.747 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:14:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:14:09.747 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:14:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:14:09.747 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:14:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:14:09.747 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:14:09.747 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:14:09.747 139328 ERROR oslo_service.periodic_task >2018-06-24 14:14:39.618 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:14:39.619 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:14:39.627 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:14:39.628 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:14:39.665 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:14:39.665 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:14:39.665 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:14:39.665 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:14:39.665 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:14:39.665 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:14:39.665 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:14:39.667 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:14:39.667 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:14:39.668 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:14:49.668 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:14:49.703 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:14:49.703 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:14:49.703 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:14:49.703 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:14:49.703 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:14:49.703 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:14:49.703 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:14:49.705 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:14:49.705 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:14:49.705 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:15:09.706 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:15:09.741 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:15:09.741 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:15:09.741 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:15:09.741 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:15:09.741 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:15:09.741 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:15:09.741 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:15:09.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:15:09.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:15:09.744 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:15:09.744 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:15:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:15:09.744 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:15:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:15:09.744 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:15:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:15:09.744 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:15:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:15:09.744 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:15:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:15:09.744 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:15:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:15:09.744 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:15:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:15:09.744 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:15:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:15:09.744 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:15:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:15:09.744 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:15:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:15:09.744 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:15:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:15:09.744 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:15:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:15:09.744 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:15:09.744 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:15:09.744 139328 ERROR oslo_service.periodic_task >2018-06-24 14:15:39.615 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:15:39.617 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:15:39.628 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:15:39.628 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:15:39.663 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:15:39.663 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:15:39.663 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:15:39.663 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:15:39.663 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:15:39.663 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:15:39.663 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:15:39.665 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:15:39.665 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:15:39.666 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:15:49.666 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:15:49.702 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:15:49.702 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:15:49.702 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:15:49.702 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:15:49.702 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:15:49.702 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:15:49.702 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:15:49.704 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:15:49.704 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:15:49.704 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:16:09.705 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:16:09.744 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:16:09.744 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:16:09.744 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:16:09.744 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:16:09.744 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:16:09.744 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:16:09.744 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:16:09.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:16:09.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.12 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:16:09.750 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:16:09.750 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:16:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:16:09.750 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:16:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:16:09.750 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:16:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:16:09.750 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:16:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:16:09.750 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:16:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:16:09.750 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:16:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:16:09.750 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:16:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:16:09.750 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:16:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:16:09.750 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:16:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:16:09.750 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:16:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:16:09.750 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:16:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:16:09.750 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:16:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:16:09.750 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:16:09.750 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:16:09.750 139328 ERROR oslo_service.periodic_task >2018-06-24 14:16:39.612 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:16:39.613 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:16:39.620 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:16:39.621 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:16:39.657 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:16:39.657 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:16:39.657 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:16:39.657 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:16:39.657 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:16:39.657 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:16:39.657 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:16:39.659 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:16:39.660 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:16:39.660 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:16:49.661 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:16:49.703 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:16:49.703 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:16:49.703 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:16:49.703 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:16:49.703 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:16:49.703 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:16:49.703 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:16:49.705 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:16:49.706 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.085 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:16:49.706 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:17:09.707 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:17:09.757 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:17:09.757 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:17:09.757 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:17:09.757 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:17:09.757 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:17:09.757 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:17:09.757 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:17:09.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:17:09.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.139 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:17:09.760 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:17:09.760 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:17:09.760 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:17:09.760 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:17:09.760 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:17:09.760 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:17:09.760 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:17:09.760 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:17:09.760 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:17:09.760 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:17:09.760 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:17:09.760 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:17:09.760 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:17:09.760 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:17:09.760 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:17:09.760 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:17:09.760 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:17:09.760 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:17:09.760 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:17:09.760 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:17:09.760 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:17:09.760 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:17:09.760 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:17:09.760 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:17:09.760 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:17:09.760 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:17:09.760 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:17:09.760 139328 ERROR oslo_service.periodic_task >2018-06-24 14:17:39.612 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:17:39.612 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:17:39.620 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:17:39.621 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:17:39.655 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:17:39.655 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:17:39.655 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:17:39.655 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:17:39.655 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:17:39.655 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:17:39.655 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:17:39.657 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:17:39.658 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:17:39.658 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:17:49.659 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:17:49.692 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:17:49.692 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:17:49.692 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:17:49.692 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:17:49.692 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:17:49.692 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:17:49.692 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:17:49.694 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:17:49.695 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:17:49.695 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:18:09.695 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:18:09.730 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:18:09.730 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:18:09.730 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:18:09.730 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:18:09.730 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:18:09.730 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:18:09.730 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:18:09.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:18:09.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:18:09.733 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:18:09.733 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:18:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:18:09.733 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:18:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:18:09.733 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:18:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:18:09.733 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:18:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:18:09.733 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:18:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:18:09.733 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:18:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:18:09.733 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:18:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:18:09.733 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:18:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:18:09.733 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:18:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:18:09.733 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:18:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:18:09.733 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:18:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:18:09.733 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:18:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:18:09.733 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:18:09.733 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:18:09.733 139328 ERROR oslo_service.periodic_task >2018-06-24 14:18:39.614 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:18:39.615 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:18:39.622 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:18:39.623 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:18:39.657 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:18:39.657 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:18:39.657 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:18:39.657 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:18:39.657 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:18:39.657 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:18:39.657 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:18:39.659 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:18:39.659 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:18:39.659 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:18:49.660 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:18:49.695 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:18:49.695 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:18:49.695 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:18:49.695 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:18:49.695 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:18:49.695 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:18:49.695 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:18:49.698 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:18:49.698 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:18:49.698 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:19:09.699 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:19:09.737 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:19:09.737 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:19:09.737 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:19:09.737 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:19:09.737 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:19:09.737 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:19:09.737 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:19:09.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:19:09.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:19:09.740 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:19:09.740 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:19:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:19:09.740 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:19:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:19:09.740 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:19:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:19:09.740 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:19:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:19:09.740 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:19:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:19:09.740 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:19:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:19:09.740 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:19:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:19:09.740 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:19:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:19:09.740 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:19:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:19:09.740 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:19:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:19:09.740 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:19:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:19:09.740 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:19:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:19:09.740 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:19:09.740 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:19:09.740 139328 ERROR oslo_service.periodic_task >2018-06-24 14:19:39.611 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:19:39.612 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:19:39.620 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:19:39.620 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:19:39.654 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:19:39.654 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:19:39.654 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:19:39.654 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:19:39.654 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:19:39.654 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:19:39.654 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:19:39.656 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:19:39.656 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:19:39.657 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:19:49.657 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:19:49.690 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:19:49.690 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:19:49.690 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:19:49.690 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:19:49.690 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:19:49.690 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:19:49.690 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:19:49.692 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:19:49.693 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:19:49.693 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:20:09.694 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:20:09.730 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:20:09.730 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:20:09.730 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:20:09.730 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:20:09.730 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:20:09.730 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:20:09.730 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:20:09.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:20:09.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:20:09.733 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:20:09.733 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:20:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:20:09.733 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:20:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:20:09.733 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:20:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:20:09.733 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:20:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:20:09.733 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:20:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:20:09.733 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:20:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:20:09.733 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:20:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:20:09.733 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:20:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:20:09.733 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:20:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:20:09.733 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:20:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:20:09.733 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:20:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:20:09.733 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:20:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:20:09.733 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:20:09.733 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:20:09.733 139328 ERROR oslo_service.periodic_task >2018-06-24 14:20:39.615 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:20:39.616 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:21:39.613 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:21:39.614 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:21:39.626 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:21:39.626 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:21:39.659 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:21:39.659 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:21:39.659 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:21:39.659 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:21:39.659 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:21:39.659 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:21:39.659 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:21:39.662 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:21:39.662 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:21:39.663 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:21:49.663 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:21:49.696 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:21:49.696 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:21:49.696 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:21:49.696 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:21:49.696 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:21:49.696 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:21:49.696 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:21:49.698 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:21:49.698 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:21:49.699 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:22:09.699 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:22:09.735 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:22:09.735 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:22:09.735 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:22:09.735 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:22:09.735 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:22:09.735 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:22:09.735 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:22:09.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:22:09.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:22:09.738 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:22:09.738 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:22:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:22:09.738 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:22:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:22:09.738 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:22:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:22:09.738 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:22:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:22:09.738 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:22:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:22:09.738 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:22:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:22:09.738 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:22:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:22:09.738 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:22:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:22:09.738 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:22:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:22:09.738 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:22:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:22:09.738 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:22:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:22:09.738 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:22:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:22:09.738 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:22:09.738 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:22:09.738 139328 ERROR oslo_service.periodic_task >2018-06-24 14:22:39.609 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:22:39.610 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:22:39.617 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:22:39.617 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:22:39.650 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:22:39.650 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:22:39.650 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:22:39.650 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:22:39.650 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:22:39.650 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:22:39.650 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:22:39.652 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:22:39.652 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:22:39.653 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:22:49.653 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:22:49.690 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:22:49.690 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:22:49.690 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:22:49.690 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:22:49.690 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:22:49.690 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:22:49.690 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:22:49.692 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:22:49.692 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:22:49.693 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:23:09.693 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:23:09.729 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:23:09.729 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:23:09.729 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:23:09.729 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:23:09.729 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:23:09.729 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:23:09.729 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:23:09.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:23:09.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:23:09.732 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:23:09.732 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:23:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:23:09.732 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:23:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:23:09.732 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:23:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:23:09.732 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:23:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:23:09.732 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:23:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:23:09.732 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:23:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:23:09.732 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:23:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:23:09.732 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:23:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:23:09.732 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:23:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:23:09.732 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:23:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:23:09.732 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:23:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:23:09.732 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:23:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:23:09.732 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:23:09.732 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:23:09.732 139328 ERROR oslo_service.periodic_task >2018-06-24 14:23:39.614 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:23:39.614 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:23:39.621 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:23:39.622 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:23:39.655 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:23:39.655 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:23:39.655 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:23:39.655 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:23:39.655 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:23:39.655 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:23:39.655 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:23:39.657 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:23:39.658 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:23:39.658 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:23:49.659 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:23:49.698 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:23:49.698 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:23:49.698 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:23:49.698 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:23:49.698 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:23:49.698 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:23:49.698 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:23:49.700 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:23:49.700 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:23:49.701 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:24:09.701 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:24:09.737 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:24:09.737 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:24:09.737 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:24:09.737 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:24:09.737 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:24:09.737 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:24:09.737 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:24:09.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:24:09.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.118 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:24:09.740 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:24:09.740 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:24:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:24:09.740 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:24:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:24:09.740 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:24:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:24:09.740 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:24:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:24:09.740 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:24:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:24:09.740 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:24:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:24:09.740 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:24:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:24:09.740 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:24:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:24:09.740 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:24:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:24:09.740 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:24:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:24:09.740 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:24:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:24:09.740 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:24:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:24:09.740 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:24:09.740 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:24:09.740 139328 ERROR oslo_service.periodic_task >2018-06-24 14:24:39.611 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:24:39.612 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:24:39.619 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:24:39.620 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:24:39.658 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:24:39.658 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:24:39.658 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:24:39.658 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:24:39.658 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:24:39.658 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:24:39.658 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:24:39.660 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:24:39.661 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:24:39.661 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:24:49.661 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:24:49.694 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:24:49.694 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:24:49.694 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:24:49.694 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:24:49.694 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:24:49.694 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:24:49.694 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:24:49.697 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:24:49.697 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:24:49.697 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:25:09.698 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:25:09.737 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:25:09.737 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:25:09.737 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:25:09.737 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:25:09.737 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:25:09.737 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:25:09.737 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:25:09.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:25:09.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.119 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:25:09.740 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:25:09.740 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:25:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:25:09.740 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:25:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:25:09.740 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:25:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:25:09.740 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:25:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:25:09.740 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:25:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:25:09.740 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:25:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:25:09.740 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:25:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:25:09.740 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:25:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:25:09.740 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:25:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:25:09.740 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:25:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:25:09.740 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:25:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:25:09.740 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:25:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:25:09.740 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:25:09.740 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:25:09.740 139328 ERROR oslo_service.periodic_task >2018-06-24 14:25:39.612 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:25:39.612 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:25:39.621 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:25:39.622 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:25:39.667 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:25:39.667 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:25:39.667 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:25:39.667 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:25:39.667 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:25:39.667 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:25:39.667 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:25:39.670 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:25:39.670 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.048 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:25:39.671 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:25:49.671 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:25:49.704 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:25:49.704 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:25:49.704 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:25:49.704 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:25:49.704 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:25:49.704 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:25:49.704 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:25:49.706 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:25:49.706 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.084 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:25:49.707 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:26:09.707 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:26:09.743 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:26:09.743 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:26:09.743 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:26:09.743 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:26:09.743 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:26:09.743 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:26:09.743 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:26:09.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:26:09.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.124 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:26:09.747 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:26:09.747 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:26:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:26:09.747 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:26:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:26:09.747 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:26:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:26:09.747 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:26:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:26:09.747 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:26:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:26:09.747 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:26:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:26:09.747 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:26:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:26:09.747 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:26:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:26:09.747 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:26:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:26:09.747 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:26:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:26:09.747 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:26:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:26:09.747 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:26:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:26:09.747 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:26:09.747 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:26:09.747 139328 ERROR oslo_service.periodic_task >2018-06-24 14:26:39.608 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:26:39.610 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:26:39.619 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:26:39.619 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:26:39.654 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:26:39.654 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:26:39.654 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:26:39.654 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:26:39.654 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:26:39.654 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:26:39.654 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:26:39.657 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:26:39.657 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:26:39.657 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:26:49.658 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:26:49.693 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:26:49.693 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:26:49.693 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:26:49.693 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:26:49.693 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:26:49.693 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:26:49.693 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:26:49.695 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:26:49.695 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:26:49.695 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:27:09.696 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:27:09.735 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:27:09.735 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:27:09.735 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:27:09.735 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:27:09.735 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:27:09.735 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:27:09.735 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:27:09.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:27:09.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:27:09.737 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:27:09.737 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:27:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:27:09.737 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:27:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:27:09.737 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:27:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:27:09.737 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:27:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:27:09.737 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:27:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:27:09.737 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:27:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:27:09.737 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:27:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:27:09.737 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:27:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:27:09.737 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:27:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:27:09.737 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:27:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:27:09.737 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:27:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:27:09.737 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:27:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:27:09.737 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:27:09.737 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:27:09.737 139328 ERROR oslo_service.periodic_task >2018-06-24 14:27:39.609 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:27:39.609 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:27:39.616 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:27:39.616 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:27:39.653 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:27:39.653 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:27:39.653 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:27:39.653 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:27:39.653 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:27:39.653 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:27:39.653 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:27:39.654 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:27:39.655 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:27:39.655 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:27:49.656 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:27:49.687 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:27:49.687 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:27:49.687 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:27:49.687 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:27:49.687 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:27:49.687 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:27:49.687 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:27:49.689 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:27:49.690 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:27:49.690 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:28:09.691 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:28:09.728 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:28:09.728 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:28:09.728 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:28:09.728 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:28:09.728 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:28:09.728 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:28:09.728 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:28:09.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:28:09.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:28:09.731 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:28:09.731 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:28:09.731 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:28:09.731 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:28:09.731 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:28:09.731 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:28:09.731 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:28:09.731 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:28:09.731 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:28:09.731 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:28:09.731 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:28:09.731 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:28:09.731 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:28:09.731 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:28:09.731 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:28:09.731 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:28:09.731 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:28:09.731 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:28:09.731 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:28:09.731 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:28:09.731 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:28:09.731 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:28:09.731 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:28:09.731 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:28:09.731 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:28:09.731 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:28:09.731 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:28:09.731 139328 ERROR oslo_service.periodic_task >2018-06-24 14:28:39.612 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:28:39.613 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:28:39.620 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:28:39.621 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:28:39.654 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:28:39.654 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:28:39.654 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:28:39.654 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:28:39.654 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:28:39.654 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:28:39.654 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:28:39.656 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:28:39.657 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:28:39.657 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:28:49.658 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:28:49.693 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:28:49.693 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:28:49.693 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:28:49.693 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:28:49.693 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:28:49.693 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:28:49.693 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:28:49.695 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:28:49.695 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:28:49.696 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:29:09.696 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:29:09.733 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:29:09.733 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:29:09.733 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:29:09.733 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:29:09.733 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:29:09.733 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:29:09.733 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:29:09.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:29:09.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:29:09.736 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:29:09.736 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:29:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:29:09.736 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:29:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:29:09.736 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:29:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:29:09.736 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:29:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:29:09.736 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:29:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:29:09.736 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:29:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:29:09.736 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:29:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:29:09.736 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:29:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:29:09.736 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:29:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:29:09.736 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:29:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:29:09.736 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:29:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:29:09.736 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:29:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:29:09.736 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:29:09.736 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:29:09.736 139328 ERROR oslo_service.periodic_task >2018-06-24 14:29:39.607 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:29:39.608 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:29:39.617 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:29:39.618 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:29:39.652 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:29:39.652 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:29:39.652 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:29:39.652 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:29:39.652 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:29:39.652 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:29:39.652 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:29:39.654 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:29:39.655 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:29:39.655 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:29:49.656 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:29:49.687 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:29:49.687 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:29:49.687 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:29:49.687 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:29:49.687 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:29:49.687 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:29:49.687 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:29:49.690 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:29:49.690 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:29:49.690 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:30:09.691 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:30:09.728 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:30:09.728 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:30:09.728 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:30:09.728 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:30:09.728 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:30:09.728 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:30:09.728 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:30:09.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:30:09.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:30:09.730 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:30:09.730 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:30:09.730 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:30:09.730 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:30:09.730 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:30:09.730 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:30:09.730 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:30:09.730 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:30:09.730 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:30:09.730 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:30:09.730 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:30:09.730 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:30:09.730 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:30:09.730 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:30:09.730 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:30:09.730 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:30:09.730 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:30:09.730 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:30:09.730 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:30:09.730 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:30:09.730 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:30:09.730 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:30:09.730 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:30:09.730 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:30:09.730 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:30:09.730 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:30:09.730 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:30:09.730 139328 ERROR oslo_service.periodic_task >2018-06-24 14:30:39.612 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:30:39.613 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:30:39.619 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:30:39.620 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:30:39.654 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:30:39.654 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:30:39.654 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:30:39.654 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:30:39.654 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:30:39.654 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:30:39.654 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:30:39.657 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:30:39.657 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:30:39.657 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:30:49.658 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:30:49.691 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:30:49.691 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:30:49.691 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:30:49.691 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:30:49.691 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:30:49.691 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:30:49.691 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:30:49.693 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:30:49.694 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:30:49.694 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:31:09.695 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:31:09.730 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:31:09.730 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:31:09.730 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:31:09.730 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:31:09.730 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:31:09.730 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:31:09.730 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:31:09.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:31:09.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:31:09.733 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:31:09.733 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:31:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:31:09.733 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:31:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:31:09.733 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:31:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:31:09.733 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:31:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:31:09.733 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:31:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:31:09.733 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:31:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:31:09.733 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:31:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:31:09.733 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:31:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:31:09.733 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:31:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:31:09.733 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:31:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:31:09.733 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:31:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:31:09.733 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:31:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:31:09.733 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:31:09.733 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:31:09.733 139328 ERROR oslo_service.periodic_task >2018-06-24 14:31:39.614 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:31:39.615 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:31:39.622 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:31:39.623 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:31:39.656 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:31:39.656 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:31:39.656 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:31:39.656 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:31:39.656 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:31:39.656 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:31:39.656 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:31:39.658 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:31:39.659 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:31:39.659 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:31:49.660 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:31:49.693 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:31:49.693 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:31:49.693 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:31:49.693 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:31:49.693 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:31:49.693 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:31:49.693 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:31:49.695 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:31:49.695 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:31:49.695 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:32:09.696 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:32:09.729 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:32:09.729 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:32:09.729 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:32:09.729 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:32:09.729 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:32:09.729 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:32:09.729 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:32:09.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:32:09.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:32:09.732 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:32:09.732 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:32:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:32:09.732 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:32:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:32:09.732 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:32:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:32:09.732 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:32:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:32:09.732 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:32:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:32:09.732 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:32:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:32:09.732 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:32:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:32:09.732 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:32:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:32:09.732 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:32:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:32:09.732 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:32:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:32:09.732 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:32:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:32:09.732 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:32:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:32:09.732 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:32:09.732 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:32:09.732 139328 ERROR oslo_service.periodic_task >2018-06-24 14:32:39.613 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:32:39.614 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:32:39.621 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:32:39.622 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:32:39.659 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:32:39.659 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:32:39.659 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:32:39.659 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:32:39.659 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:32:39.659 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:32:39.659 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:32:39.663 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:32:39.663 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.041 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:32:39.664 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:32:49.664 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:32:49.698 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:32:49.698 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:32:49.698 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:32:49.698 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:32:49.698 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:32:49.698 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:32:49.698 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:32:49.700 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:32:49.700 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:32:49.701 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:33:09.701 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:33:09.735 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:33:09.735 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:33:09.735 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:33:09.735 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:33:09.735 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:33:09.735 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:33:09.735 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:33:09.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:33:09.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:33:09.738 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:33:09.738 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:33:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:33:09.738 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:33:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:33:09.738 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:33:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:33:09.738 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:33:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:33:09.738 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:33:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:33:09.738 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:33:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:33:09.738 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:33:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:33:09.738 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:33:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:33:09.738 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:33:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:33:09.738 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:33:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:33:09.738 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:33:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:33:09.738 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:33:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:33:09.738 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:33:09.738 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:33:09.738 139328 ERROR oslo_service.periodic_task >2018-06-24 14:33:39.609 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:33:39.610 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:33:39.619 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:33:39.620 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:33:39.656 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:33:39.656 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:33:39.656 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:33:39.656 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:33:39.656 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:33:39.656 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:33:39.656 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:33:39.658 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:33:39.658 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:33:39.659 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:33:49.659 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:33:49.691 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:33:49.691 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:33:49.691 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:33:49.691 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:33:49.691 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:33:49.691 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:33:49.691 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:33:49.693 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:33:49.693 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:33:49.694 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:34:09.694 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:34:09.732 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:34:09.732 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:34:09.732 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:34:09.732 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:34:09.732 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:34:09.732 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:34:09.732 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:34:09.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:34:09.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:34:09.735 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:34:09.735 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:34:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:34:09.735 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:34:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:34:09.735 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:34:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:34:09.735 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:34:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:34:09.735 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:34:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:34:09.735 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:34:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:34:09.735 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:34:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:34:09.735 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:34:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:34:09.735 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:34:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:34:09.735 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:34:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:34:09.735 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:34:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:34:09.735 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:34:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:34:09.735 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:34:09.735 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:34:09.735 139328 ERROR oslo_service.periodic_task >2018-06-24 14:34:39.606 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:34:39.607 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:34:39.619 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:34:39.619 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:34:39.657 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:34:39.657 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:34:39.657 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:34:39.657 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:34:39.657 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:34:39.657 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:34:39.657 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:34:39.659 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:34:39.659 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:34:39.659 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:34:49.660 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:34:49.696 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:34:49.696 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:34:49.696 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:34:49.696 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:34:49.696 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:34:49.696 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:34:49.696 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:34:49.698 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:34:49.699 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:34:49.699 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:35:09.699 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:35:09.733 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:35:09.733 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:35:09.733 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:35:09.733 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:35:09.733 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:35:09.733 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:35:09.733 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:35:09.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:35:09.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:35:09.735 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:35:09.735 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:35:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:35:09.735 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:35:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:35:09.735 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:35:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:35:09.735 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:35:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:35:09.735 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:35:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:35:09.735 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:35:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:35:09.735 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:35:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:35:09.735 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:35:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:35:09.735 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:35:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:35:09.735 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:35:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:35:09.735 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:35:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:35:09.735 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:35:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:35:09.735 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:35:09.735 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:35:09.735 139328 ERROR oslo_service.periodic_task >2018-06-24 14:35:39.607 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:35:39.607 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:35:39.616 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:35:39.617 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:35:39.650 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:35:39.650 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:35:39.650 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:35:39.650 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:35:39.650 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:35:39.650 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:35:39.650 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:35:39.652 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:35:39.652 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:35:39.652 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:35:49.653 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:35:49.688 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:35:49.688 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:35:49.688 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:35:49.688 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:35:49.688 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:35:49.688 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:35:49.688 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:35:49.690 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:35:49.691 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:35:49.691 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:36:09.691 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:36:09.727 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:36:09.727 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:36:09.727 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:36:09.727 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:36:09.727 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:36:09.727 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:36:09.727 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:36:09.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:36:09.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:36:09.730 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:36:09.730 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:36:09.730 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:36:09.730 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:36:09.730 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:36:09.730 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:36:09.730 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:36:09.730 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:36:09.730 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:36:09.730 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:36:09.730 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:36:09.730 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:36:09.730 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:36:09.730 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:36:09.730 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:36:09.730 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:36:09.730 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:36:09.730 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:36:09.730 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:36:09.730 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:36:09.730 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:36:09.730 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:36:09.730 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:36:09.730 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:36:09.730 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:36:09.730 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:36:09.730 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:36:09.730 139328 ERROR oslo_service.periodic_task >2018-06-24 14:36:39.612 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:36:39.613 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:36:39.619 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:36:39.620 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:36:39.653 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:36:39.653 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:36:39.653 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:36:39.653 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:36:39.653 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:36:39.653 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:36:39.653 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:36:39.655 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:36:39.656 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:36:39.656 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:36:49.656 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:36:49.691 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:36:49.691 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:36:49.691 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:36:49.691 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:36:49.691 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:36:49.691 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:36:49.691 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:36:49.693 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:36:49.694 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:36:49.694 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:37:09.695 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:37:09.730 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:37:09.730 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:37:09.730 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:37:09.730 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:37:09.730 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:37:09.730 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:37:09.730 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:37:09.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:37:09.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:37:09.733 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:37:09.733 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:37:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:37:09.733 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:37:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:37:09.733 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:37:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:37:09.733 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:37:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:37:09.733 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:37:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:37:09.733 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:37:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:37:09.733 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:37:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:37:09.733 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:37:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:37:09.733 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:37:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:37:09.733 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:37:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:37:09.733 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:37:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:37:09.733 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:37:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:37:09.733 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:37:09.733 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:37:09.733 139328 ERROR oslo_service.periodic_task >2018-06-24 14:37:39.615 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:37:39.616 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:37:39.624 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:37:39.625 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:37:39.659 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:37:39.659 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:37:39.659 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:37:39.659 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:37:39.659 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:37:39.659 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:37:39.659 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:37:39.661 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:37:39.661 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:37:39.662 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:37:49.662 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:37:49.699 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:37:49.699 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:37:49.699 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:37:49.699 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:37:49.699 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:37:49.699 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:37:49.699 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:37:49.701 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:37:49.701 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:37:49.702 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:38:09.702 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:38:09.737 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:38:09.737 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:38:09.737 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:38:09.737 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:38:09.737 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:38:09.737 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:38:09.737 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:38:09.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:38:09.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:38:09.740 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:38:09.740 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:38:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:38:09.740 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:38:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:38:09.740 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:38:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:38:09.740 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:38:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:38:09.740 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:38:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:38:09.740 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:38:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:38:09.740 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:38:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:38:09.740 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:38:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:38:09.740 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:38:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:38:09.740 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:38:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:38:09.740 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:38:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:38:09.740 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:38:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:38:09.740 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:38:09.740 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:38:09.740 139328 ERROR oslo_service.periodic_task >2018-06-24 14:38:39.611 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:38:39.612 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:38:39.619 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:38:39.620 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:38:39.665 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:38:39.665 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:38:39.665 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:38:39.665 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:38:39.665 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:38:39.665 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:38:39.665 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:38:39.667 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:38:39.668 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.048 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:38:39.668 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:38:49.669 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:38:49.704 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:38:49.704 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:38:49.704 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:38:49.704 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:38:49.704 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:38:49.704 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:38:49.704 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:38:49.706 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:38:49.706 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.086 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:38:49.706 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:39:09.707 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:39:09.754 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:39:09.754 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:39:09.754 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:39:09.754 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:39:09.754 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:39:09.754 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:39:09.754 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:39:09.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:39:09.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.137 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:39:09.757 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:39:09.757 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:39:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:39:09.757 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:39:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:39:09.757 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:39:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:39:09.757 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:39:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:39:09.757 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:39:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:39:09.757 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:39:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:39:09.757 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:39:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:39:09.757 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:39:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:39:09.757 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:39:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:39:09.757 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:39:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:39:09.757 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:39:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:39:09.757 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:39:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:39:09.757 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:39:09.757 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:39:09.757 139328 ERROR oslo_service.periodic_task >2018-06-24 14:39:39.608 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:39:39.609 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:39:39.617 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:39:39.618 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:39:39.657 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:39:39.657 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:39:39.657 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:39:39.657 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:39:39.657 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:39:39.657 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:39:39.657 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:39:39.660 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:39:39.660 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.042 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:39:39.660 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:39:49.661 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:39:49.696 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:39:49.696 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:39:49.696 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:39:49.696 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:39:49.696 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:39:49.696 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:39:49.696 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:39:49.698 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:39:49.698 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.081 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:39:49.699 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:40:09.699 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:40:09.740 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:40:09.740 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:40:09.740 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:40:09.740 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:40:09.740 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:40:09.740 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:40:09.740 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:40:09.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:40:09.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.125 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:40:09.743 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:40:09.743 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:40:09.743 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:40:09.743 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:40:09.743 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:40:09.743 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:40:09.743 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:40:09.743 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:40:09.743 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:40:09.743 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:40:09.743 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:40:09.743 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:40:09.743 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:40:09.743 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:40:09.743 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:40:09.743 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:40:09.743 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:40:09.743 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:40:09.743 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:40:09.743 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:40:09.743 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:40:09.743 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:40:09.743 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:40:09.743 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:40:09.743 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:40:09.743 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:40:09.743 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:40:09.743 139328 ERROR oslo_service.periodic_task >2018-06-24 14:40:39.605 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:40:39.605 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:40:39.613 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:40:39.613 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:40:39.653 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:40:39.653 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:40:39.653 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:40:39.653 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:40:39.653 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:40:39.653 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:40:39.653 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:40:39.655 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:40:39.656 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.042 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:40:39.656 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:40:49.656 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:40:49.692 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:40:49.692 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:40:49.692 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:40:49.692 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:40:49.692 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:40:49.692 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:40:49.692 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:40:49.694 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:40:49.694 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.081 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:40:49.695 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:41:09.695 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:41:09.729 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:41:09.729 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:41:09.729 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:41:09.729 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:41:09.729 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:41:09.729 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:41:09.729 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:41:09.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:41:09.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.118 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:41:09.732 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:41:09.732 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:41:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:41:09.732 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:41:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:41:09.732 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:41:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:41:09.732 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:41:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:41:09.732 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:41:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:41:09.732 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:41:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:41:09.732 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:41:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:41:09.732 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:41:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:41:09.732 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:41:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:41:09.732 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:41:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:41:09.732 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:41:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:41:09.732 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:41:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:41:09.732 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:41:09.732 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:41:09.732 139328 ERROR oslo_service.periodic_task >2018-06-24 14:41:39.603 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:41:39.604 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:41:39.612 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:41:39.612 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:41:39.659 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:41:39.659 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:41:39.659 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:41:39.659 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:41:39.659 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:41:39.659 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:41:39.659 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:41:39.662 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:41:39.662 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.049 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:41:39.663 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:41:49.664 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:41:49.698 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:41:49.698 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:41:49.698 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:41:49.698 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:41:49.698 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:41:49.698 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:41:49.698 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:41:49.700 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:41:49.700 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.087 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:41:49.700 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:42:09.701 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:42:09.738 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:42:09.738 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:42:09.738 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:42:09.738 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:42:09.738 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:42:09.738 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:42:09.738 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:42:09.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:42:09.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.127 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:42:09.741 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:42:09.741 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:42:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:42:09.741 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:42:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:42:09.741 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:42:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:42:09.741 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:42:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:42:09.741 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:42:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:42:09.741 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:42:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:42:09.741 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:42:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:42:09.741 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:42:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:42:09.741 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:42:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:42:09.741 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:42:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:42:09.741 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:42:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:42:09.741 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:42:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:42:09.741 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:42:09.741 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:42:09.741 139328 ERROR oslo_service.periodic_task >2018-06-24 14:42:39.602 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:42:39.602 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:42:39.610 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:42:39.610 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:42:39.646 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:42:39.646 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:42:39.646 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:42:39.646 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:42:39.646 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:42:39.646 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:42:39.646 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:42:39.649 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:42:39.649 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:42:39.650 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:42:49.651 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:42:49.686 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:42:49.686 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:42:49.686 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:42:49.686 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:42:49.686 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:42:49.686 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:42:49.686 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:42:49.688 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:42:49.689 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:42:49.689 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:43:09.689 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:43:09.724 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:43:09.724 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:43:09.724 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:43:09.724 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:43:09.724 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:43:09.724 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:43:09.724 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:43:09.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:43:09.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:43:09.727 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:43:09.727 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:43:09.727 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:43:09.727 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:43:09.727 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:43:09.727 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:43:09.727 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:43:09.727 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:43:09.727 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:43:09.727 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:43:09.727 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:43:09.727 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:43:09.727 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:43:09.727 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:43:09.727 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:43:09.727 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:43:09.727 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:43:09.727 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:43:09.727 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:43:09.727 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:43:09.727 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:43:09.727 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:43:09.727 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:43:09.727 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:43:09.727 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:43:09.727 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:43:09.727 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:43:09.727 139328 ERROR oslo_service.periodic_task >2018-06-24 14:43:39.599 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:43:39.599 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:43:39.606 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:43:39.607 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:43:39.639 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:43:39.639 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:43:39.639 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:43:39.639 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:43:39.639 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:43:39.639 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:43:39.639 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:43:39.640 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:43:39.641 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:43:39.641 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:43:49.641 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:43:49.673 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:43:49.673 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:43:49.673 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:43:49.673 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:43:49.673 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:43:49.673 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:43:49.673 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:43:49.675 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:43:49.676 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.069 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:43:49.676 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:44:09.677 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:44:09.715 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:44:09.715 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:44:09.715 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:44:09.715 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:44:09.715 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:44:09.715 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:44:09.715 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:44:09.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:44:09.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:44:09.719 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:44:09.719 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:44:09.719 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:44:09.719 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:44:09.719 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:44:09.719 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:44:09.719 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:44:09.719 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:44:09.719 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:44:09.719 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:44:09.719 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:44:09.719 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:44:09.719 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:44:09.719 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:44:09.719 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:44:09.719 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:44:09.719 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:44:09.719 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:44:09.719 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:44:09.719 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:44:09.719 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:44:09.719 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:44:09.719 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:44:09.719 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:44:09.719 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:44:09.719 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:44:09.719 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:44:09.719 139328 ERROR oslo_service.periodic_task >2018-06-24 14:44:39.601 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:44:39.601 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:44:39.608 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:44:39.609 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:44:39.642 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:44:39.642 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:44:39.642 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:44:39.642 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:44:39.642 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:44:39.642 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:44:39.642 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:44:39.644 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:44:39.644 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:44:39.644 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:44:49.645 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:44:49.681 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:44:49.681 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:44:49.681 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:44:49.681 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:44:49.681 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:44:49.681 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:44:49.681 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:44:49.683 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:44:49.683 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:44:49.683 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:45:09.684 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:45:09.721 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:45:09.721 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:45:09.721 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:45:09.721 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:45:09.721 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:45:09.721 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:45:09.721 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:45:09.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:45:09.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:45:09.724 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:45:09.724 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:45:09.724 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:45:09.724 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:45:09.724 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:45:09.724 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:45:09.724 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:45:09.724 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:45:09.724 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:45:09.724 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:45:09.724 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:45:09.724 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:45:09.724 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:45:09.724 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:45:09.724 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:45:09.724 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:45:09.724 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:45:09.724 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:45:09.724 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:45:09.724 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:45:09.724 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:45:09.724 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:45:09.724 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:45:09.724 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:45:09.724 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:45:09.724 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:45:09.724 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:45:09.724 139328 ERROR oslo_service.periodic_task >2018-06-24 14:45:39.605 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:45:39.606 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:45:39.613 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:45:39.614 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:45:39.646 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:45:39.646 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:45:39.646 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:45:39.646 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:45:39.646 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:45:39.646 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:45:39.646 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:45:39.648 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:45:39.649 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:45:39.649 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:45:49.649 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:45:49.682 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:45:49.682 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:45:49.682 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:45:49.682 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:45:49.682 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:45:49.682 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:45:49.682 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:45:49.685 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:45:49.685 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:45:49.685 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:46:09.686 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:46:09.721 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:46:09.721 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:46:09.721 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:46:09.721 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:46:09.721 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:46:09.721 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:46:09.721 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:46:09.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:46:09.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:46:09.723 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:46:09.723 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:46:09.723 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:46:09.723 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:46:09.723 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:46:09.723 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:46:09.723 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:46:09.723 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:46:09.723 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:46:09.723 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:46:09.723 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:46:09.723 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:46:09.723 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:46:09.723 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:46:09.723 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:46:09.723 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:46:09.723 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:46:09.723 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:46:09.723 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:46:09.723 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:46:09.723 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:46:09.723 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:46:09.723 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:46:09.723 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:46:09.723 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:46:09.723 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:46:09.723 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:46:09.723 139328 ERROR oslo_service.periodic_task >2018-06-24 14:46:39.605 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:46:39.606 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:46:39.619 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:46:39.619 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:46:39.654 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:46:39.654 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:46:39.654 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:46:39.654 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:46:39.654 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:46:39.654 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:46:39.654 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:46:39.657 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:46:39.657 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:46:39.657 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:46:49.658 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:46:49.697 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:46:49.697 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:46:49.697 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:46:49.697 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:46:49.697 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:46:49.697 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:46:49.697 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:46:49.700 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:46:49.700 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.08 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:46:49.700 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:47:09.701 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:47:09.737 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:47:09.737 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:47:09.737 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:47:09.737 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:47:09.737 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:47:09.737 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:47:09.737 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:47:09.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:47:09.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.12 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:47:09.740 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:47:09.740 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:47:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:47:09.740 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:47:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:47:09.740 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:47:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:47:09.740 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:47:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:47:09.740 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:47:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:47:09.740 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:47:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:47:09.740 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:47:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:47:09.740 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:47:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:47:09.740 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:47:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:47:09.740 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:47:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:47:09.740 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:47:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:47:09.740 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:47:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:47:09.740 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:47:09.740 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:47:09.740 139328 ERROR oslo_service.periodic_task >2018-06-24 14:47:39.601 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:47:39.602 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:47:39.609 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:47:39.609 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:47:39.642 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:47:39.642 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:47:39.642 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:47:39.642 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:47:39.642 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:47:39.642 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:47:39.642 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:47:39.645 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:47:39.646 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:47:39.646 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:47:49.647 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:47:49.686 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:47:49.686 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:47:49.686 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:47:49.686 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:47:49.686 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:47:49.686 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:47:49.686 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:47:49.688 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:47:49.689 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:47:49.689 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:48:09.690 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:48:09.724 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:48:09.724 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:48:09.724 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:48:09.724 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:48:09.724 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:48:09.724 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:48:09.724 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:48:09.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:48:09.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:48:09.727 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:48:09.727 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:48:09.727 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:48:09.727 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:48:09.727 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:48:09.727 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:48:09.727 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:48:09.727 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:48:09.727 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:48:09.727 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:48:09.727 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:48:09.727 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:48:09.727 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:48:09.727 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:48:09.727 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:48:09.727 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:48:09.727 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:48:09.727 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:48:09.727 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:48:09.727 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:48:09.727 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:48:09.727 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:48:09.727 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:48:09.727 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:48:09.727 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:48:09.727 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:48:09.727 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:48:09.727 139328 ERROR oslo_service.periodic_task >2018-06-24 14:48:39.599 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:48:39.599 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:48:39.606 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:48:39.606 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:48:39.640 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:48:39.640 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:48:39.640 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:48:39.640 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:48:39.640 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:48:39.640 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:48:39.640 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:48:39.642 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:48:39.642 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:48:39.642 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:48:49.643 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:48:49.679 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:48:49.679 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:48:49.679 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:48:49.679 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:48:49.679 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:48:49.679 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:48:49.679 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:48:49.682 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:48:49.682 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:48:49.682 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:49:09.683 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:49:09.716 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:49:09.716 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:49:09.716 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:49:09.716 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:49:09.716 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:49:09.716 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:49:09.716 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:49:09.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:49:09.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:49:09.719 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:49:09.719 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:49:09.719 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:49:09.719 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:49:09.719 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:49:09.719 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:49:09.719 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:49:09.719 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:49:09.719 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:49:09.719 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:49:09.719 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:49:09.719 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:49:09.719 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:49:09.719 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:49:09.719 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:49:09.719 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:49:09.719 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:49:09.719 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:49:09.719 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:49:09.719 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:49:09.719 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:49:09.719 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:49:09.719 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:49:09.719 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:49:09.719 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:49:09.719 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:49:09.719 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:49:09.719 139328 ERROR oslo_service.periodic_task >2018-06-24 14:49:39.600 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:49:39.601 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:49:39.608 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:49:39.608 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:49:39.642 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:49:39.642 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:49:39.642 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:49:39.642 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:49:39.642 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:49:39.642 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:49:39.642 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:49:39.644 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:49:39.645 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:49:39.645 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:49:49.645 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:49:49.684 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:49:49.684 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:49:49.684 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:49:49.684 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:49:49.684 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:49:49.684 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:49:49.684 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:49:49.686 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:49:49.686 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:49:49.686 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:50:09.687 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:50:09.722 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:50:09.722 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:50:09.722 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:50:09.722 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:50:09.722 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:50:09.722 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:50:09.722 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:50:09.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:50:09.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:50:09.725 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:50:09.725 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:50:09.725 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:50:09.725 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:50:09.725 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:50:09.725 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:50:09.725 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:50:09.725 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:50:09.725 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:50:09.725 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:50:09.725 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:50:09.725 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:50:09.725 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:50:09.725 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:50:09.725 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:50:09.725 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:50:09.725 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:50:09.725 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:50:09.725 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:50:09.725 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:50:09.725 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:50:09.725 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:50:09.725 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:50:09.725 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:50:09.725 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:50:09.725 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:50:09.725 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:50:09.725 139328 ERROR oslo_service.periodic_task >2018-06-24 14:50:39.596 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:50:39.597 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:50:39.608 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:50:39.609 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:50:39.647 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:50:39.647 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:50:39.647 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:50:39.647 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:50:39.647 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:50:39.647 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:50:39.647 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:50:39.649 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:50:39.650 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.041 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:50:39.650 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:50:49.651 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:50:49.683 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:50:49.683 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:50:49.683 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:50:49.683 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:50:49.683 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:50:49.683 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:50:49.683 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:50:49.685 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:50:49.686 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:50:49.686 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:51:09.686 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:51:09.719 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:51:09.719 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:51:09.719 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:51:09.719 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:51:09.719 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:51:09.719 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:51:09.719 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:51:09.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:51:09.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:51:09.722 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:51:09.722 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:51:09.722 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:51:09.722 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:51:09.722 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:51:09.722 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:51:09.722 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:51:09.722 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:51:09.722 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:51:09.722 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:51:09.722 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:51:09.722 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:51:09.722 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:51:09.722 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:51:09.722 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:51:09.722 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:51:09.722 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:51:09.722 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:51:09.722 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:51:09.722 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:51:09.722 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:51:09.722 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:51:09.722 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:51:09.722 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:51:09.722 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:51:09.722 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:51:09.722 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:51:09.722 139328 ERROR oslo_service.periodic_task >2018-06-24 14:51:39.594 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:51:39.594 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:51:39.601 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:51:39.602 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:51:39.635 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:51:39.635 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:51:39.635 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:51:39.635 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:51:39.635 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:51:39.635 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:51:39.635 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:51:39.637 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:51:39.638 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:51:39.638 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:51:49.639 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:51:49.674 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:51:49.674 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:51:49.674 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:51:49.674 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:51:49.674 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:51:49.674 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:51:49.674 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:51:49.676 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:51:49.677 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:51:49.677 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:52:09.677 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:52:09.712 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:52:09.712 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:52:09.712 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:52:09.712 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:52:09.712 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:52:09.712 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:52:09.712 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:52:09.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:52:09.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:52:09.715 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:52:09.715 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:52:09.715 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:52:09.715 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:52:09.715 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:52:09.715 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:52:09.715 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:52:09.715 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:52:09.715 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:52:09.715 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:52:09.715 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:52:09.715 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:52:09.715 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:52:09.715 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:52:09.715 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:52:09.715 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:52:09.715 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:52:09.715 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:52:09.715 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:52:09.715 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:52:09.715 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:52:09.715 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:52:09.715 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:52:09.715 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:52:09.715 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:52:09.715 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:52:09.715 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:52:09.715 139328 ERROR oslo_service.periodic_task >2018-06-24 14:52:39.596 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:52:39.597 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:52:39.604 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:52:39.604 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:52:39.640 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:52:39.640 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:52:39.640 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:52:39.640 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:52:39.640 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:52:39.640 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:52:39.640 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:52:39.642 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:52:39.643 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:52:39.643 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:52:49.644 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:52:49.677 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:52:49.677 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:52:49.677 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:52:49.677 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:52:49.677 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:52:49.677 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:52:49.677 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:52:49.679 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:52:49.679 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:52:49.679 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:53:09.680 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:53:09.713 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:53:09.713 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:53:09.713 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:53:09.713 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:53:09.713 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:53:09.713 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:53:09.713 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:53:09.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:53:09.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:53:09.717 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:53:09.717 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:53:09.717 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:53:09.717 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:53:09.717 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:53:09.717 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:53:09.717 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:53:09.717 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:53:09.717 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:53:09.717 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:53:09.717 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:53:09.717 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:53:09.717 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:53:09.717 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:53:09.717 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:53:09.717 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:53:09.717 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:53:09.717 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:53:09.717 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:53:09.717 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:53:09.717 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:53:09.717 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:53:09.717 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:53:09.717 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:53:09.717 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:53:09.717 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:53:09.717 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:53:09.717 139328 ERROR oslo_service.periodic_task >2018-06-24 14:53:39.598 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:53:39.598 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:53:39.605 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:53:39.606 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:53:39.641 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:53:39.641 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:53:39.641 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:53:39.641 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:53:39.641 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:53:39.641 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:53:39.641 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:53:39.643 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:53:39.644 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:53:39.644 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:53:49.644 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:53:49.677 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:53:49.677 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:53:49.677 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:53:49.677 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:53:49.677 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:53:49.677 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:53:49.677 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:53:49.679 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:53:49.680 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:53:49.680 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:54:09.680 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:54:09.716 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:54:09.716 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:54:09.716 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:54:09.716 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:54:09.716 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:54:09.716 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:54:09.716 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:54:09.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:54:09.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:54:09.720 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:54:09.720 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:54:09.720 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:54:09.720 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:54:09.720 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:54:09.720 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:54:09.720 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:54:09.720 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:54:09.720 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:54:09.720 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:54:09.720 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:54:09.720 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:54:09.720 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:54:09.720 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:54:09.720 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:54:09.720 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:54:09.720 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:54:09.720 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:54:09.720 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:54:09.720 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:54:09.720 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:54:09.720 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:54:09.720 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:54:09.720 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:54:09.720 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:54:09.720 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:54:09.720 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:54:09.720 139328 ERROR oslo_service.periodic_task >2018-06-24 14:54:39.602 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:54:39.603 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:54:39.610 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:54:39.610 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:54:39.644 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:54:39.644 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:54:39.644 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:54:39.644 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:54:39.644 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:54:39.644 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:54:39.644 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:54:39.647 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:54:39.647 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:54:39.648 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:54:49.648 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:54:49.681 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:54:49.681 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:54:49.681 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:54:49.681 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:54:49.681 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:54:49.681 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:54:49.681 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:54:49.682 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:54:49.683 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:54:49.683 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:55:09.684 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:55:09.722 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:55:09.722 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:55:09.722 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:55:09.722 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:55:09.722 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:55:09.722 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:55:09.722 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:55:09.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:55:09.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:55:09.725 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:55:09.725 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:55:09.725 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:55:09.725 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:55:09.725 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:55:09.725 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:55:09.725 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:55:09.725 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:55:09.725 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:55:09.725 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:55:09.725 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:55:09.725 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:55:09.725 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:55:09.725 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:55:09.725 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:55:09.725 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:55:09.725 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:55:09.725 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:55:09.725 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:55:09.725 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:55:09.725 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:55:09.725 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:55:09.725 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:55:09.725 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:55:09.725 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:55:09.725 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:55:09.725 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:55:09.725 139328 ERROR oslo_service.periodic_task >2018-06-24 14:55:39.597 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:55:39.598 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:55:39.605 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:55:39.606 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:55:39.640 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:55:39.640 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:55:39.640 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:55:39.640 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:55:39.640 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:55:39.640 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:55:39.640 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:55:39.643 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:55:39.643 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:55:39.644 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:55:49.644 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:55:49.677 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:55:49.677 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:55:49.677 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:55:49.677 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:55:49.677 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:55:49.677 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:55:49.677 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:55:49.679 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:55:49.679 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:55:49.679 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:56:09.680 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:56:09.716 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:56:09.716 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:56:09.716 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:56:09.716 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:56:09.716 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:56:09.716 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:56:09.716 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:56:09.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:56:09.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:56:09.718 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:56:09.718 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:56:09.718 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:56:09.718 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:56:09.718 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:56:09.718 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:56:09.718 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:56:09.718 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:56:09.718 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:56:09.718 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:56:09.718 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:56:09.718 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:56:09.718 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:56:09.718 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:56:09.718 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:56:09.718 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:56:09.718 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:56:09.718 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:56:09.718 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:56:09.718 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:56:09.718 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:56:09.718 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:56:09.718 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:56:09.718 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:56:09.718 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:56:09.718 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:56:09.718 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:56:09.718 139328 ERROR oslo_service.periodic_task >2018-06-24 14:56:39.600 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:56:39.601 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:56:39.608 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:56:39.609 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:56:39.644 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:56:39.644 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:56:39.644 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:56:39.644 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:56:39.644 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:56:39.644 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:56:39.644 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:56:39.646 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:56:39.646 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:56:39.646 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:56:49.647 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:56:49.680 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:56:49.680 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:56:49.680 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:56:49.680 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:56:49.680 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:56:49.680 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:56:49.680 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:56:49.682 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:56:49.682 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:56:49.683 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:57:09.683 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:57:09.718 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:57:09.718 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:57:09.718 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:57:09.718 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:57:09.718 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:57:09.718 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:57:09.718 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:57:09.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:57:09.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:57:09.721 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:57:09.721 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:57:09.721 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:57:09.721 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:57:09.721 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:57:09.721 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:57:09.721 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:57:09.721 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:57:09.721 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:57:09.721 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:57:09.721 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:57:09.721 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:57:09.721 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:57:09.721 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:57:09.721 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:57:09.721 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:57:09.721 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:57:09.721 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:57:09.721 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:57:09.721 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:57:09.721 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:57:09.721 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:57:09.721 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:57:09.721 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:57:09.721 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:57:09.721 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:57:09.721 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:57:09.721 139328 ERROR oslo_service.periodic_task >2018-06-24 14:57:39.603 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:57:39.604 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:57:39.614 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:57:39.615 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:57:39.648 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:57:39.648 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:57:39.648 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:57:39.648 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:57:39.648 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:57:39.648 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:57:39.648 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:57:39.650 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:57:39.650 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:57:39.651 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:57:49.651 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:57:49.684 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:57:49.684 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:57:49.684 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:57:49.684 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:57:49.684 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:57:49.684 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:57:49.684 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:57:49.687 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:57:49.688 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:57:49.688 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:58:09.689 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:58:09.722 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:58:09.722 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:58:09.722 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:58:09.722 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:58:09.722 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:58:09.722 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:58:09.722 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:58:09.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:58:09.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:58:09.725 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:58:09.725 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:58:09.725 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:58:09.725 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:58:09.725 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:58:09.725 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:58:09.725 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:58:09.725 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:58:09.725 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:58:09.725 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:58:09.725 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:58:09.725 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:58:09.725 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:58:09.725 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:58:09.725 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:58:09.725 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:58:09.725 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:58:09.725 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:58:09.725 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:58:09.725 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:58:09.725 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:58:09.725 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:58:09.725 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:58:09.725 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:58:09.725 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:58:09.725 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:58:09.725 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:58:09.725 139328 ERROR oslo_service.periodic_task >2018-06-24 14:58:39.606 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:58:39.607 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:58:39.614 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:58:39.615 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:58:39.652 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:58:39.652 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:58:39.652 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:58:39.652 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:58:39.652 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:58:39.652 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:58:39.652 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:58:39.654 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:58:39.654 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:58:39.655 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:58:49.655 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:58:49.693 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:58:49.693 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:58:49.693 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:58:49.693 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:58:49.693 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:58:49.693 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:58:49.693 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:58:49.695 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:58:49.695 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:58:49.695 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:59:09.696 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:59:09.731 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:59:09.731 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:59:09.731 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:59:09.731 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:59:09.731 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:59:09.731 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:59:09.731 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:59:09.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:59:09.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.118 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:59:09.734 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:59:09.734 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 14:59:09.734 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 14:59:09.734 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 14:59:09.734 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 14:59:09.734 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 14:59:09.734 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 14:59:09.734 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 14:59:09.734 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 14:59:09.734 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 14:59:09.734 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 14:59:09.734 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 14:59:09.734 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 14:59:09.734 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 14:59:09.734 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 14:59:09.734 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 14:59:09.734 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 14:59:09.734 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 14:59:09.734 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 14:59:09.734 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 14:59:09.734 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 14:59:09.734 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 14:59:09.734 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 14:59:09.734 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 14:59:09.734 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 14:59:09.734 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 14:59:09.734 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 14:59:09.734 139328 ERROR oslo_service.periodic_task >2018-06-24 14:59:39.606 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:59:39.607 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 14:59:39.614 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 14:59:39.615 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:59:39.651 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:59:39.651 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:59:39.651 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:59:39.651 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:59:39.651 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:59:39.651 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:59:39.651 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:59:39.654 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:59:39.655 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:59:39.655 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 14:59:49.656 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 14:59:49.689 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 14:59:49.689 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 14:59:49.689 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 14:59:49.689 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 14:59:49.689 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 14:59:49.689 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 14:59:49.689 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 14:59:49.692 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 14:59:49.692 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 14:59:49.693 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:00:09.694 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:00:09.730 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:00:09.730 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:00:09.730 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:00:09.730 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:00:09.730 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:00:09.730 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:00:09.730 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:00:09.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:00:09.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.118 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:00:09.733 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:00:09.733 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:00:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:00:09.733 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:00:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:00:09.733 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:00:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:00:09.733 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:00:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:00:09.733 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:00:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:00:09.733 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:00:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:00:09.733 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:00:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:00:09.733 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:00:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:00:09.733 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:00:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:00:09.733 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:00:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:00:09.733 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:00:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:00:09.733 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:00:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:00:09.733 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:00:09.733 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:00:09.733 139328 ERROR oslo_service.periodic_task >2018-06-24 15:00:39.604 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:00:39.605 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:00:39.613 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:00:39.614 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:00:39.652 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:00:39.652 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:00:39.652 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:00:39.652 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:00:39.652 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:00:39.652 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:00:39.652 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:00:39.655 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:00:39.655 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.041 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:00:39.656 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:00:49.656 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:00:49.689 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:00:49.689 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:00:49.689 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:00:49.689 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:00:49.689 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:00:49.689 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:00:49.689 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:00:49.691 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:00:49.692 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:00:49.692 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:01:09.692 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:01:09.740 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:01:09.740 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:01:09.740 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:01:09.740 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:01:09.740 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:01:09.740 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:01:09.740 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:01:09.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:01:09.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.13 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:01:09.744 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:01:09.744 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:01:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:01:09.744 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:01:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:01:09.744 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:01:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:01:09.744 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:01:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:01:09.744 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:01:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:01:09.744 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:01:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:01:09.744 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:01:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:01:09.744 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:01:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:01:09.744 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:01:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:01:09.744 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:01:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:01:09.744 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:01:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:01:09.744 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:01:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:01:09.744 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:01:09.744 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:01:09.744 139328 ERROR oslo_service.periodic_task >2018-06-24 15:01:39.606 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:01:39.608 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:01:39.614 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:01:39.614 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:01:39.650 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:01:39.650 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:01:39.650 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:01:39.650 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:01:39.650 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:01:39.650 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:01:39.650 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:01:39.652 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:01:39.652 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:01:39.653 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:01:49.653 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:01:49.686 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:01:49.686 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:01:49.686 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:01:49.686 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:01:49.686 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:01:49.686 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:01:49.686 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:01:49.688 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:01:49.688 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:01:49.688 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:02:09.689 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:02:09.723 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:02:09.723 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:02:09.723 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:02:09.723 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:02:09.723 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:02:09.723 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:02:09.723 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:02:09.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:02:09.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:02:09.726 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:02:09.726 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:02:09.726 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:02:09.726 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:02:09.726 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:02:09.726 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:02:09.726 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:02:09.726 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:02:09.726 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:02:09.726 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:02:09.726 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:02:09.726 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:02:09.726 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:02:09.726 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:02:09.726 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:02:09.726 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:02:09.726 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:02:09.726 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:02:09.726 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:02:09.726 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:02:09.726 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:02:09.726 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:02:09.726 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:02:09.726 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:02:09.726 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:02:09.726 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:02:09.726 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:02:09.726 139328 ERROR oslo_service.periodic_task >2018-06-24 15:02:39.607 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:02:39.608 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:02:39.616 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:02:39.616 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:02:39.651 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:02:39.651 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:02:39.651 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:02:39.651 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:02:39.651 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:02:39.651 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:02:39.651 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:02:39.653 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:02:39.653 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:02:39.654 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:02:49.654 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:02:49.690 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:02:49.690 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:02:49.690 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:02:49.690 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:02:49.690 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:02:49.690 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:02:49.690 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:02:49.691 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:02:49.692 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:02:49.692 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:03:09.693 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:03:09.731 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:03:09.731 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:03:09.731 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:03:09.731 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:03:09.731 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:03:09.731 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:03:09.731 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:03:09.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:03:09.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.118 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:03:09.735 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:03:09.735 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:03:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:03:09.735 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:03:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:03:09.735 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:03:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:03:09.735 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:03:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:03:09.735 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:03:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:03:09.735 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:03:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:03:09.735 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:03:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:03:09.735 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:03:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:03:09.735 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:03:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:03:09.735 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:03:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:03:09.735 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:03:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:03:09.735 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:03:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:03:09.735 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:03:09.735 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:03:09.735 139328 ERROR oslo_service.periodic_task >2018-06-24 15:03:39.607 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:03:39.608 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:03:39.616 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:03:39.616 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:03:39.658 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:03:39.658 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:03:39.658 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:03:39.658 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:03:39.658 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:03:39.658 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:03:39.658 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:03:39.661 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:03:39.661 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.044 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:03:39.661 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:03:49.662 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:03:49.697 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:03:49.697 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:03:49.697 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:03:49.697 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:03:49.697 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:03:49.697 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:03:49.697 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:03:49.699 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:03:49.699 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.083 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:03:49.700 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:04:09.700 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:04:09.736 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:04:09.736 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:04:09.736 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:04:09.736 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:04:09.736 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:04:09.736 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:04:09.736 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:04:09.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:04:09.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.122 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:04:09.739 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:04:09.739 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:04:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:04:09.739 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:04:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:04:09.739 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:04:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:04:09.739 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:04:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:04:09.739 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:04:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:04:09.739 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:04:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:04:09.739 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:04:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:04:09.739 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:04:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:04:09.739 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:04:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:04:09.739 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:04:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:04:09.739 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:04:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:04:09.739 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:04:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:04:09.739 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:04:09.739 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:04:09.739 139328 ERROR oslo_service.periodic_task >2018-06-24 15:04:39.611 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:04:39.612 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:04:39.627 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:04:39.628 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:04:39.683 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:04:39.683 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:04:39.683 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:04:39.683 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:04:39.683 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:04:39.683 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:04:39.683 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:04:39.688 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:04:39.689 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.061 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:04:39.689 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:04:49.690 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:04:49.723 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:04:49.723 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:04:49.723 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:04:49.723 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:04:49.723 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:04:49.723 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:04:49.723 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:04:49.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:04:49.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.098 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:04:49.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:05:09.727 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:05:09.760 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:05:09.760 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:05:09.760 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:05:09.760 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:05:09.760 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:05:09.760 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:05:09.760 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:05:09.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:05:09.764 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.136 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:05:09.764 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:05:09.764 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:05:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:05:09.764 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:05:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:05:09.764 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:05:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:05:09.764 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:05:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:05:09.764 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:05:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:05:09.764 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:05:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:05:09.764 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:05:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:05:09.764 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:05:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:05:09.764 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:05:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:05:09.764 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:05:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:05:09.764 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:05:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:05:09.764 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:05:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:05:09.764 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:05:09.764 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:05:09.764 139328 ERROR oslo_service.periodic_task >2018-06-24 15:05:39.606 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:05:39.607 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:05:39.616 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:05:39.617 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:05:39.658 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:05:39.658 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:05:39.658 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:05:39.658 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:05:39.658 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:05:39.658 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:05:39.658 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:05:39.660 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:05:39.660 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.044 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:05:39.661 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:05:49.661 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:05:49.695 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:05:49.695 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:05:49.695 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:05:49.695 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:05:49.695 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:05:49.695 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:05:49.695 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:05:49.697 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:05:49.697 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.081 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:05:49.698 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:06:09.698 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:06:09.733 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:06:09.733 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:06:09.733 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:06:09.733 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:06:09.733 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:06:09.733 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:06:09.733 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:06:09.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:06:09.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.119 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:06:09.736 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:06:09.736 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:06:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:06:09.736 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:06:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:06:09.736 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:06:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:06:09.736 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:06:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:06:09.736 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:06:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:06:09.736 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:06:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:06:09.736 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:06:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:06:09.736 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:06:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:06:09.736 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:06:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:06:09.736 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:06:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:06:09.736 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:06:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:06:09.736 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:06:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:06:09.736 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:06:09.736 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:06:09.736 139328 ERROR oslo_service.periodic_task >2018-06-24 15:06:39.607 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:06:39.608 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:06:39.616 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:06:39.616 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:06:39.650 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:06:39.650 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:06:39.650 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:06:39.650 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:06:39.650 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:06:39.650 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:06:39.650 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:06:39.653 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:06:39.653 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:06:39.654 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:06:49.655 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:06:49.693 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:06:49.693 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:06:49.693 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:06:49.693 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:06:49.693 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:06:49.693 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:06:49.693 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:06:49.696 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:06:49.696 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:06:49.696 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:07:09.697 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:07:09.736 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:07:09.736 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:07:09.736 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:07:09.736 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:07:09.736 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:07:09.736 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:07:09.736 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:07:09.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:07:09.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.123 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:07:09.740 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:07:09.740 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:07:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:07:09.740 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:07:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:07:09.740 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:07:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:07:09.740 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:07:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:07:09.740 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:07:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:07:09.740 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:07:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:07:09.740 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:07:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:07:09.740 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:07:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:07:09.740 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:07:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:07:09.740 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:07:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:07:09.740 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:07:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:07:09.740 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:07:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:07:09.740 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:07:09.740 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:07:09.740 139328 ERROR oslo_service.periodic_task >2018-06-24 15:07:39.611 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:07:39.612 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:07:39.618 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:07:39.619 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:07:39.653 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:07:39.653 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:07:39.653 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:07:39.653 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:07:39.653 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:07:39.653 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:07:39.653 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:07:39.655 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:07:39.655 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:07:39.655 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:07:49.656 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:07:49.696 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:07:49.696 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:07:49.696 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:07:49.696 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:07:49.696 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:07:49.696 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:07:49.696 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:07:49.698 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:07:49.698 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:07:49.698 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:08:09.699 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:08:09.731 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:08:09.731 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:08:09.731 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:08:09.731 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:08:09.731 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:08:09.731 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:08:09.731 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:08:09.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:08:09.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:08:09.735 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:08:09.735 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:08:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:08:09.735 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:08:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:08:09.735 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:08:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:08:09.735 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:08:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:08:09.735 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:08:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:08:09.735 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:08:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:08:09.735 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:08:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:08:09.735 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:08:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:08:09.735 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:08:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:08:09.735 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:08:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:08:09.735 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:08:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:08:09.735 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:08:09.735 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:08:09.735 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:08:09.735 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:08:09.735 139328 ERROR oslo_service.periodic_task >2018-06-24 15:08:39.616 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:08:39.617 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:08:39.632 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:08:39.633 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:08:39.667 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:08:39.667 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:08:39.667 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:08:39.667 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:08:39.667 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:08:39.667 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:08:39.667 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:08:39.669 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:08:39.669 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:08:39.669 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:08:49.670 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:08:49.704 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:08:49.704 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:08:49.704 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:08:49.704 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:08:49.704 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:08:49.704 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:08:49.704 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:08:49.706 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:08:49.706 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:08:49.707 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:09:09.707 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:09:09.742 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:09:09.742 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:09:09.742 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:09:09.742 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:09:09.742 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:09:09.742 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:09:09.742 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:09:09.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:09:09.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:09:09.745 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:09:09.745 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:09:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:09:09.745 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:09:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:09:09.745 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:09:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:09:09.745 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:09:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:09:09.745 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:09:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:09:09.745 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:09:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:09:09.745 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:09:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:09:09.745 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:09:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:09:09.745 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:09:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:09:09.745 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:09:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:09:09.745 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:09:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:09:09.745 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:09:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:09:09.745 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:09:09.745 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:09:09.745 139328 ERROR oslo_service.periodic_task >2018-06-24 15:09:39.616 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:09:39.617 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:09:39.624 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:09:39.625 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:09:39.657 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:09:39.657 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:09:39.657 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:09:39.657 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:09:39.657 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:09:39.657 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:09:39.657 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:09:39.659 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:09:39.660 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:09:39.660 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:09:49.661 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:09:49.692 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:09:49.692 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:09:49.692 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:09:49.692 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:09:49.692 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:09:49.692 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:09:49.692 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:09:49.695 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:09:49.695 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.07 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:09:49.696 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:10:09.696 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:10:09.730 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:10:09.730 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:10:09.730 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:10:09.730 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:10:09.730 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:10:09.730 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:10:09.730 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:10:09.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:10:09.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:10:09.734 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:10:09.734 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:10:09.734 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:10:09.734 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:10:09.734 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:10:09.734 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:10:09.734 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:10:09.734 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:10:09.734 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:10:09.734 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:10:09.734 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:10:09.734 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:10:09.734 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:10:09.734 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:10:09.734 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:10:09.734 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:10:09.734 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:10:09.734 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:10:09.734 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:10:09.734 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:10:09.734 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:10:09.734 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:10:09.734 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:10:09.734 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:10:09.734 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:10:09.734 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:10:09.734 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:10:09.734 139328 ERROR oslo_service.periodic_task >2018-06-24 15:10:39.616 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:10:39.616 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:10:39.622 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:10:39.623 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:10:39.657 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:10:39.657 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:10:39.657 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:10:39.657 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:10:39.657 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:10:39.657 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:10:39.657 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:10:39.658 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:10:39.659 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:10:39.659 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:10:49.660 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:10:49.692 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:10:49.692 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:10:49.692 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:10:49.692 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:10:49.692 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:10:49.692 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:10:49.692 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:10:49.694 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:10:49.694 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:10:49.695 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:11:09.695 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:11:09.730 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:11:09.730 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:11:09.730 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:11:09.730 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:11:09.730 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:11:09.730 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:11:09.730 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:11:09.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:11:09.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:11:09.733 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:11:09.733 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:11:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:11:09.733 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:11:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:11:09.733 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:11:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:11:09.733 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:11:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:11:09.733 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:11:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:11:09.733 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:11:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:11:09.733 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:11:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:11:09.733 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:11:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:11:09.733 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:11:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:11:09.733 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:11:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:11:09.733 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:11:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:11:09.733 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:11:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:11:09.733 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:11:09.733 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:11:09.733 139328 ERROR oslo_service.periodic_task >2018-06-24 15:11:39.614 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:11:39.615 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:11:39.623 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:11:39.624 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:11:39.656 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:11:39.656 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:11:39.656 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:11:39.656 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:11:39.656 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:11:39.656 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:11:39.656 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:11:39.658 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:11:39.659 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:11:39.659 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:11:49.659 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:11:49.694 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:11:49.694 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:11:49.694 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:11:49.694 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:11:49.694 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:11:49.694 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:11:49.694 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:11:49.697 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:11:49.697 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:11:49.697 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:12:09.698 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:12:09.730 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:12:09.730 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:12:09.730 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:12:09.730 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:12:09.730 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:12:09.730 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:12:09.730 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:12:09.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:12:09.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:12:09.733 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:12:09.733 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:12:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:12:09.733 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:12:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:12:09.733 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:12:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:12:09.733 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:12:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:12:09.733 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:12:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:12:09.733 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:12:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:12:09.733 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:12:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:12:09.733 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:12:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:12:09.733 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:12:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:12:09.733 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:12:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:12:09.733 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:12:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:12:09.733 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:12:09.733 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:12:09.733 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:12:09.733 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:12:09.733 139328 ERROR oslo_service.periodic_task >2018-06-24 15:12:39.614 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:12:39.615 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:12:39.656 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:12:39.656 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:12:39.656 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:12:39.656 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:12:39.656 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:12:39.656 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:12:39.656 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:12:39.658 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:12:39.658 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.042 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:12:39.658 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:12:49.659 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:12:49.695 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:12:49.695 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:12:49.695 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:12:49.695 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:12:49.695 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:12:49.695 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:12:49.695 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:12:49.697 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:12:49.698 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.082 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:12:49.698 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:13:09.699 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:13:09.734 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:13:09.734 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:13:09.734 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:13:09.734 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:13:09.734 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:13:09.734 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:13:09.734 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:13:09.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:13:09.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.121 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:13:09.737 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:13:09.737 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:13:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:13:09.737 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:13:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:13:09.737 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:13:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:13:09.737 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:13:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:13:09.737 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:13:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:13:09.737 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:13:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:13:09.737 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:13:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:13:09.737 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:13:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:13:09.737 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:13:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:13:09.737 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:13:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:13:09.737 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:13:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:13:09.737 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:13:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:13:09.737 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:13:09.737 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:13:09.737 139328 ERROR oslo_service.periodic_task >2018-06-24 15:13:39.618 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:13:39.619 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:13:39.627 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:13:39.627 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:13:39.664 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:13:39.664 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:13:39.664 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:13:39.664 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:13:39.664 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:13:39.664 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:13:39.664 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:13:39.666 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:13:39.666 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:13:39.667 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:13:49.667 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:13:49.701 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:13:49.701 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:13:49.701 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:13:49.701 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:13:49.701 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:13:49.701 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:13:49.701 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:13:49.703 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:13:49.703 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:13:49.703 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:14:09.704 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:14:09.737 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:14:09.737 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:14:09.737 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:14:09.737 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:14:09.737 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:14:09.737 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:14:09.737 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:14:09.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:14:09.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:14:09.739 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:14:09.739 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:14:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:14:09.739 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:14:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:14:09.739 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:14:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:14:09.739 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:14:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:14:09.739 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:14:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:14:09.739 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:14:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:14:09.739 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:14:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:14:09.739 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:14:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:14:09.739 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:14:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:14:09.739 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:14:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:14:09.739 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:14:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:14:09.739 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:14:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:14:09.739 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:14:09.739 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:14:09.739 139328 ERROR oslo_service.periodic_task >2018-06-24 15:14:39.621 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:14:39.621 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:14:39.629 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:14:39.629 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:14:39.681 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:14:39.681 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:14:39.681 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:14:39.681 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:14:39.681 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:14:39.681 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:14:39.681 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:14:39.684 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:14:39.684 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.054 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:14:39.685 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:14:49.685 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:14:49.717 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:14:49.717 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:14:49.717 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:14:49.717 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:14:49.717 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:14:49.717 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:14:49.717 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:14:49.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:14:49.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.09 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:14:49.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:15:09.721 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:15:09.754 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:15:09.754 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:15:09.754 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:15:09.754 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:15:09.754 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:15:09.754 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:15:09.754 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:15:09.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:15:09.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.126 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:15:09.756 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:15:09.756 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:15:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:15:09.756 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:15:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:15:09.756 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:15:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:15:09.756 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:15:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:15:09.756 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:15:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:15:09.756 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:15:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:15:09.756 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:15:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:15:09.756 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:15:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:15:09.756 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:15:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:15:09.756 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:15:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:15:09.756 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:15:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:15:09.756 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:15:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:15:09.756 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:15:09.756 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:15:09.756 139328 ERROR oslo_service.periodic_task >2018-06-24 15:15:39.618 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:15:39.619 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:16:39.617 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:16:39.618 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:16:39.625 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:16:39.625 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:16:39.662 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:16:39.662 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:16:39.662 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:16:39.662 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:16:39.662 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:16:39.662 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:16:39.662 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:16:39.665 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:16:39.665 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:16:39.665 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:16:49.666 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:16:49.698 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:16:49.698 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:16:49.698 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:16:49.698 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:16:49.698 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:16:49.698 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:16:49.698 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:16:49.700 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:16:49.700 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:16:49.701 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:17:09.701 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:17:09.733 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:17:09.733 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:17:09.733 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:17:09.733 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:17:09.733 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:17:09.733 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:17:09.733 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:17:09.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:17:09.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:17:09.736 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:17:09.736 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:17:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:17:09.736 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:17:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:17:09.736 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:17:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:17:09.736 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:17:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:17:09.736 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:17:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:17:09.736 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:17:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:17:09.736 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:17:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:17:09.736 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:17:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:17:09.736 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:17:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:17:09.736 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:17:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:17:09.736 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:17:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:17:09.736 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:17:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:17:09.736 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:17:09.736 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:17:09.736 139328 ERROR oslo_service.periodic_task >2018-06-24 15:17:39.617 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:17:39.618 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:17:39.629 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:17:39.629 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:17:39.665 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:17:39.665 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:17:39.665 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:17:39.665 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:17:39.665 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:17:39.665 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:17:39.665 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:17:39.668 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:17:39.668 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:17:39.669 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:17:49.670 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:17:49.702 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:17:49.702 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:17:49.702 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:17:49.702 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:17:49.702 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:17:49.702 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:17:49.702 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:17:49.704 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:17:49.705 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:17:49.705 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:18:09.706 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:18:09.740 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:18:09.740 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:18:09.740 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:18:09.740 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:18:09.740 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:18:09.740 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:18:09.740 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:18:09.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:18:09.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:18:09.743 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:18:09.743 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:18:09.743 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:18:09.743 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:18:09.743 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:18:09.743 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:18:09.743 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:18:09.743 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:18:09.743 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:18:09.743 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:18:09.743 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:18:09.743 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:18:09.743 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:18:09.743 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:18:09.743 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:18:09.743 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:18:09.743 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:18:09.743 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:18:09.743 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:18:09.743 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:18:09.743 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:18:09.743 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:18:09.743 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:18:09.743 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:18:09.743 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:18:09.743 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:18:09.743 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:18:09.743 139328 ERROR oslo_service.periodic_task >2018-06-24 15:18:39.615 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:18:39.616 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:18:39.623 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:18:39.624 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:18:39.663 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:18:39.663 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:18:39.663 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:18:39.663 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:18:39.663 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:18:39.663 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:18:39.663 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:18:39.665 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:18:39.665 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.041 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:18:39.665 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:18:49.666 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:18:49.699 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:18:49.699 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:18:49.699 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:18:49.699 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:18:49.699 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:18:49.699 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:18:49.699 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:18:49.702 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:18:49.702 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:18:49.702 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:19:09.703 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:19:09.740 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:19:09.740 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:19:09.740 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:19:09.740 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:19:09.740 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:19:09.740 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:19:09.740 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:19:09.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:19:09.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.118 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:19:09.743 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:19:09.743 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:19:09.743 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:19:09.743 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:19:09.743 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:19:09.743 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:19:09.743 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:19:09.743 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:19:09.743 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:19:09.743 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:19:09.743 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:19:09.743 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:19:09.743 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:19:09.743 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:19:09.743 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:19:09.743 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:19:09.743 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:19:09.743 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:19:09.743 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:19:09.743 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:19:09.743 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:19:09.743 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:19:09.743 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:19:09.743 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:19:09.743 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:19:09.743 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:19:09.743 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:19:09.743 139328 ERROR oslo_service.periodic_task >2018-06-24 15:19:39.614 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:19:39.615 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:19:39.623 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:19:39.623 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:19:39.658 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:19:39.658 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:19:39.658 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:19:39.658 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:19:39.658 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:19:39.658 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:19:39.658 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:19:39.660 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:19:39.660 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:19:39.661 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:19:49.661 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:19:49.697 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:19:49.697 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:19:49.697 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:19:49.697 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:19:49.697 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:19:49.697 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:19:49.697 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:19:49.699 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:19:49.699 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:19:49.699 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:20:09.700 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:20:09.735 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:20:09.735 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:20:09.735 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:20:09.735 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:20:09.735 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:20:09.735 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:20:09.735 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:20:09.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:20:09.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:20:09.738 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:20:09.738 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:20:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:20:09.738 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:20:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:20:09.738 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:20:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:20:09.738 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:20:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:20:09.738 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:20:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:20:09.738 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:20:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:20:09.738 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:20:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:20:09.738 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:20:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:20:09.738 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:20:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:20:09.738 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:20:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:20:09.738 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:20:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:20:09.738 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:20:09.738 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:20:09.738 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:20:09.738 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:20:09.738 139328 ERROR oslo_service.periodic_task >2018-06-24 15:20:39.609 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:20:39.610 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:20:39.619 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:20:39.619 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:20:39.652 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:20:39.652 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:20:39.652 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:20:39.652 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:20:39.652 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:20:39.652 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:20:39.652 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:20:39.655 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:20:39.655 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:20:39.656 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:20:49.656 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:20:49.690 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:20:49.690 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:20:49.690 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:20:49.690 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:20:49.690 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:20:49.690 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:20:49.690 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:20:49.692 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:20:49.692 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:20:49.692 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:21:09.693 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:21:09.728 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:21:09.728 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:21:09.728 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:21:09.728 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:21:09.728 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:21:09.728 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:21:09.728 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:21:09.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:21:09.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:21:09.731 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:21:09.731 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:21:09.731 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:21:09.731 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:21:09.731 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:21:09.731 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:21:09.731 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:21:09.731 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:21:09.731 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:21:09.731 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:21:09.731 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:21:09.731 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:21:09.731 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:21:09.731 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:21:09.731 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:21:09.731 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:21:09.731 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:21:09.731 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:21:09.731 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:21:09.731 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:21:09.731 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:21:09.731 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:21:09.731 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:21:09.731 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:21:09.731 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:21:09.731 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:21:09.731 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:21:09.731 139328 ERROR oslo_service.periodic_task >2018-06-24 15:21:39.612 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:21:39.613 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:21:39.620 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:21:39.621 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:21:39.660 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:21:39.660 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:21:39.660 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:21:39.660 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:21:39.660 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:21:39.660 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:21:39.660 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:21:39.663 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:21:39.663 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.043 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:21:39.664 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:21:49.664 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:21:49.704 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:21:49.704 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:21:49.704 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:21:49.704 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:21:49.704 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:21:49.704 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:21:49.704 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:21:49.706 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:21:49.706 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.085 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:21:49.707 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:22:09.707 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:22:09.749 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:22:09.749 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:22:09.749 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:22:09.749 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:22:09.749 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:22:09.749 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:22:09.749 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:22:09.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:22:09.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.131 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:22:09.752 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:22:09.752 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:22:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:22:09.752 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:22:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:22:09.752 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:22:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:22:09.752 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:22:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:22:09.752 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:22:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:22:09.752 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:22:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:22:09.752 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:22:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:22:09.752 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:22:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:22:09.752 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:22:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:22:09.752 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:22:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:22:09.752 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:22:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:22:09.752 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:22:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:22:09.752 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:22:09.752 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:22:09.752 139328 ERROR oslo_service.periodic_task >2018-06-24 15:22:39.613 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:22:39.615 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:22:39.621 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:22:39.621 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:22:39.654 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:22:39.654 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:22:39.654 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:22:39.654 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:22:39.654 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:22:39.654 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:22:39.654 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:22:39.656 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:22:39.657 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:22:39.657 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:22:49.657 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:22:49.697 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:22:49.697 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:22:49.697 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:22:49.697 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:22:49.697 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:22:49.697 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:22:49.697 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:22:49.699 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:22:49.699 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:22:49.700 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:23:09.700 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:23:09.733 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:23:09.733 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:23:09.733 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:23:09.733 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:23:09.733 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:23:09.733 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:23:09.733 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:23:09.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:23:09.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:23:09.736 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:23:09.736 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:23:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:23:09.736 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:23:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:23:09.736 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:23:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:23:09.736 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:23:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:23:09.736 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:23:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:23:09.736 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:23:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:23:09.736 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:23:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:23:09.736 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:23:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:23:09.736 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:23:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:23:09.736 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:23:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:23:09.736 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:23:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:23:09.736 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:23:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:23:09.736 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:23:09.736 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:23:09.736 139328 ERROR oslo_service.periodic_task >2018-06-24 15:23:39.617 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:23:39.618 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:23:39.626 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:23:39.627 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:23:39.661 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:23:39.661 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:23:39.661 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:23:39.661 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:23:39.661 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:23:39.661 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:23:39.661 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:23:39.663 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:23:39.663 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:23:39.664 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:23:49.664 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:23:49.700 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:23:49.700 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:23:49.700 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:23:49.700 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:23:49.700 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:23:49.700 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:23:49.700 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:23:49.703 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:23:49.703 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:23:49.703 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:24:09.704 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:24:09.738 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:24:09.738 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:24:09.738 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:24:09.738 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:24:09.738 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:24:09.738 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:24:09.738 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:24:09.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:24:09.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:24:09.741 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:24:09.741 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:24:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:24:09.741 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:24:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:24:09.741 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:24:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:24:09.741 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:24:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:24:09.741 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:24:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:24:09.741 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:24:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:24:09.741 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:24:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:24:09.741 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:24:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:24:09.741 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:24:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:24:09.741 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:24:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:24:09.741 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:24:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:24:09.741 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:24:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:24:09.741 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:24:09.741 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:24:09.741 139328 ERROR oslo_service.periodic_task >2018-06-24 15:24:39.622 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:24:39.623 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:24:39.631 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:24:39.632 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:24:39.668 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:24:39.668 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:24:39.668 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:24:39.668 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:24:39.668 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:24:39.668 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:24:39.668 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:24:39.670 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:24:39.670 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:24:39.671 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:24:49.671 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:24:49.715 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:24:49.715 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:24:49.715 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:24:49.715 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:24:49.715 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:24:49.715 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:24:49.715 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:24:49.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:24:49.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.086 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:24:49.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:25:09.718 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:25:09.752 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:25:09.752 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:25:09.752 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:25:09.752 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:25:09.752 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:25:09.752 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:25:09.752 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:25:09.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:25:09.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.123 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:25:09.755 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:25:09.755 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:25:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:25:09.755 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:25:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:25:09.755 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:25:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:25:09.755 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:25:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:25:09.755 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:25:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:25:09.755 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:25:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:25:09.755 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:25:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:25:09.755 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:25:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:25:09.755 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:25:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:25:09.755 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:25:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:25:09.755 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:25:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:25:09.755 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:25:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:25:09.755 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:25:09.755 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:25:09.755 139328 ERROR oslo_service.periodic_task >2018-06-24 15:25:39.626 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:25:39.627 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:25:39.633 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:25:39.634 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:25:39.667 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:25:39.667 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:25:39.667 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:25:39.667 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:25:39.667 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:25:39.667 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:25:39.667 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:25:39.669 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:25:39.670 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:25:39.670 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:25:49.671 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:25:49.706 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:25:49.706 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:25:49.706 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:25:49.706 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:25:49.706 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:25:49.706 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:25:49.706 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:25:49.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:25:49.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.08 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:25:49.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:26:09.715 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:26:09.749 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:26:09.749 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:26:09.749 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:26:09.749 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:26:09.749 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:26:09.749 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:26:09.749 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:26:09.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:26:09.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.118 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:26:09.752 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:26:09.752 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:26:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:26:09.752 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:26:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:26:09.752 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:26:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:26:09.752 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:26:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:26:09.752 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:26:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:26:09.752 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:26:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:26:09.752 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:26:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:26:09.752 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:26:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:26:09.752 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:26:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:26:09.752 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:26:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:26:09.752 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:26:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:26:09.752 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:26:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:26:09.752 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:26:09.752 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:26:09.752 139328 ERROR oslo_service.periodic_task >2018-06-24 15:26:39.623 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:26:39.624 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:26:39.631 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:26:39.631 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:26:39.664 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:26:39.664 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:26:39.664 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:26:39.664 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:26:39.664 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:26:39.664 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:26:39.664 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:26:39.666 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:26:39.667 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:26:39.667 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:26:49.667 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:26:49.707 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:26:49.707 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:26:49.707 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:26:49.707 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:26:49.707 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:26:49.707 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:26:49.707 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:26:49.710 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:26:49.710 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:26:49.710 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:27:09.711 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:27:09.745 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:27:09.745 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:27:09.745 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:27:09.745 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:27:09.745 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:27:09.745 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:27:09.745 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:27:09.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:27:09.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:27:09.748 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:27:09.748 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:27:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:27:09.748 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:27:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:27:09.748 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:27:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:27:09.748 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:27:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:27:09.748 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:27:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:27:09.748 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:27:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:27:09.748 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:27:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:27:09.748 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:27:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:27:09.748 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:27:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:27:09.748 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:27:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:27:09.748 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:27:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:27:09.748 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:27:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:27:09.748 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:27:09.748 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:27:09.748 139328 ERROR oslo_service.periodic_task >2018-06-24 15:27:39.619 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:27:39.621 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:27:39.629 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:27:39.629 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:27:39.664 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:27:39.664 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:27:39.664 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:27:39.664 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:27:39.664 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:27:39.664 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:27:39.664 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:27:39.666 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:27:39.666 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:27:39.666 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:27:49.667 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:27:49.701 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:27:49.701 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:27:49.701 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:27:49.701 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:27:49.701 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:27:49.701 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:27:49.701 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:27:49.703 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:27:49.703 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:27:49.704 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:28:09.704 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:28:09.741 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:28:09.741 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:28:09.741 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:28:09.741 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:28:09.741 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:28:09.741 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:28:09.741 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:28:09.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:28:09.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:28:09.744 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:28:09.744 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:28:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:28:09.744 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:28:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:28:09.744 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:28:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:28:09.744 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:28:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:28:09.744 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:28:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:28:09.744 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:28:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:28:09.744 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:28:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:28:09.744 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:28:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:28:09.744 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:28:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:28:09.744 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:28:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:28:09.744 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:28:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:28:09.744 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:28:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:28:09.744 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:28:09.744 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:28:09.744 139328 ERROR oslo_service.periodic_task >2018-06-24 15:28:39.615 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:28:39.616 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:28:39.626 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:28:39.627 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:28:39.662 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:28:39.662 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:28:39.662 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:28:39.662 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:28:39.662 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:28:39.662 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:28:39.662 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:28:39.665 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:28:39.665 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:28:39.665 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:28:49.666 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:28:49.698 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:28:49.698 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:28:49.698 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:28:49.698 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:28:49.698 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:28:49.698 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:28:49.698 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:28:49.700 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:28:49.700 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:28:49.701 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:29:09.701 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:29:09.735 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:29:09.735 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:29:09.735 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:29:09.735 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:29:09.735 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:29:09.735 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:29:09.735 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:29:09.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:29:09.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:29:09.739 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:29:09.739 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:29:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:29:09.739 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:29:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:29:09.739 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:29:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:29:09.739 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:29:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:29:09.739 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:29:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:29:09.739 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:29:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:29:09.739 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:29:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:29:09.739 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:29:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:29:09.739 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:29:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:29:09.739 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:29:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:29:09.739 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:29:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:29:09.739 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:29:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:29:09.739 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:29:09.739 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:29:09.739 139328 ERROR oslo_service.periodic_task >2018-06-24 15:29:39.620 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:29:39.621 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:29:39.628 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:29:39.628 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:29:39.661 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:29:39.661 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:29:39.661 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:29:39.661 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:29:39.661 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:29:39.661 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:29:39.661 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:29:39.664 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:29:39.664 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:29:39.665 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:29:49.665 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:29:49.699 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:29:49.699 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:29:49.699 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:29:49.699 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:29:49.699 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:29:49.699 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:29:49.699 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:29:49.701 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:29:49.701 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:29:49.702 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:30:09.702 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:30:09.736 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:30:09.736 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:30:09.736 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:30:09.736 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:30:09.736 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:30:09.736 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:30:09.736 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:30:09.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:30:09.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:30:09.739 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:30:09.739 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:30:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:30:09.739 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:30:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:30:09.739 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:30:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:30:09.739 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:30:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:30:09.739 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:30:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:30:09.739 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:30:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:30:09.739 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:30:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:30:09.739 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:30:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:30:09.739 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:30:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:30:09.739 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:30:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:30:09.739 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:30:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:30:09.739 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:30:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:30:09.739 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:30:09.739 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:30:09.739 139328 ERROR oslo_service.periodic_task >2018-06-24 15:30:39.620 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:30:39.621 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:30:39.628 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:30:39.628 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:30:39.665 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:30:39.665 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:30:39.665 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:30:39.665 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:30:39.665 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:30:39.665 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:30:39.665 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:30:39.668 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:30:39.668 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:30:39.668 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:30:49.669 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:30:49.701 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:30:49.701 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:30:49.701 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:30:49.701 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:30:49.701 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:30:49.701 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:30:49.701 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:30:49.703 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:30:49.703 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:30:49.703 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:31:09.704 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:31:09.741 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:31:09.741 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:31:09.741 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:31:09.741 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:31:09.741 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:31:09.741 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:31:09.741 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:31:09.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:31:09.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:31:09.744 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:31:09.744 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:31:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:31:09.744 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:31:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:31:09.744 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:31:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:31:09.744 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:31:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:31:09.744 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:31:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:31:09.744 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:31:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:31:09.744 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:31:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:31:09.744 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:31:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:31:09.744 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:31:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:31:09.744 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:31:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:31:09.744 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:31:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:31:09.744 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:31:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:31:09.744 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:31:09.744 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:31:09.744 139328 ERROR oslo_service.periodic_task >2018-06-24 15:31:39.616 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:31:39.617 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:31:39.624 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:31:39.625 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:31:39.661 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:31:39.661 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:31:39.661 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:31:39.661 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:31:39.661 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:31:39.661 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:31:39.661 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:31:39.663 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:31:39.664 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:31:39.664 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:31:49.665 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:31:49.700 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:31:49.700 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:31:49.700 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:31:49.700 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:31:49.700 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:31:49.700 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:31:49.700 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:31:49.702 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:31:49.703 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:31:49.703 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:32:09.703 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:32:09.736 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:32:09.736 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:32:09.736 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:32:09.736 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:32:09.736 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:32:09.736 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:32:09.736 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:32:09.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:32:09.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:32:09.739 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:32:09.739 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:32:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:32:09.739 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:32:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:32:09.739 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:32:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:32:09.739 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:32:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:32:09.739 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:32:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:32:09.739 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:32:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:32:09.739 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:32:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:32:09.739 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:32:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:32:09.739 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:32:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:32:09.739 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:32:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:32:09.739 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:32:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:32:09.739 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:32:09.739 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:32:09.739 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:32:09.739 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:32:09.739 139328 ERROR oslo_service.periodic_task >2018-06-24 15:32:39.612 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:32:39.612 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:32:39.620 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:32:39.621 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:32:39.660 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:32:39.660 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:32:39.660 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:32:39.660 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:32:39.660 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:32:39.660 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:32:39.660 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:32:39.662 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:32:39.663 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.042 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:32:39.663 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:32:49.664 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:32:49.697 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:32:49.697 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:32:49.697 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:32:49.697 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:32:49.697 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:32:49.697 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:32:49.697 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:32:49.700 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:32:49.700 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:32:49.700 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:33:09.701 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:33:09.734 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:33:09.734 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:33:09.734 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:33:09.734 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:33:09.734 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:33:09.734 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:33:09.734 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:33:09.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:33:09.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:33:09.736 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:33:09.736 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:33:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:33:09.736 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:33:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:33:09.736 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:33:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:33:09.736 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:33:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:33:09.736 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:33:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:33:09.736 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:33:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:33:09.736 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:33:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:33:09.736 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:33:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:33:09.736 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:33:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:33:09.736 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:33:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:33:09.736 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:33:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:33:09.736 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:33:09.736 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:33:09.736 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:33:09.736 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:33:09.736 139328 ERROR oslo_service.periodic_task >2018-06-24 15:33:39.608 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:33:39.609 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:33:39.617 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:33:39.617 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:33:39.654 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:33:39.654 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:33:39.654 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:33:39.654 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:33:39.654 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:33:39.654 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:33:39.654 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:33:39.656 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:33:39.657 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:33:39.657 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:33:49.658 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:33:49.692 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:33:49.692 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:33:49.692 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:33:49.692 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:33:49.692 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:33:49.692 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:33:49.692 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:33:49.694 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:33:49.694 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:33:49.694 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:34:09.695 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:34:09.744 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:34:09.744 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:34:09.744 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:34:09.744 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:34:09.744 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:34:09.744 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:34:09.744 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:34:09.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:34:09.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.13 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:34:09.748 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:34:09.748 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:34:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:34:09.748 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:34:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:34:09.748 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:34:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:34:09.748 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:34:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:34:09.748 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:34:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:34:09.748 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:34:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:34:09.748 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:34:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:34:09.748 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:34:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:34:09.748 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:34:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:34:09.748 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:34:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:34:09.748 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:34:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:34:09.748 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:34:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:34:09.748 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:34:09.748 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:34:09.748 139328 ERROR oslo_service.periodic_task >2018-06-24 15:34:39.609 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:34:39.610 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:34:39.617 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:34:39.618 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:34:39.655 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:34:39.655 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:34:39.655 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:34:39.655 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:34:39.655 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:34:39.655 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:34:39.655 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:34:39.658 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:34:39.658 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:34:39.659 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:34:49.659 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:34:49.693 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:34:49.693 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:34:49.693 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:34:49.693 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:34:49.693 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:34:49.693 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:34:49.693 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:34:49.695 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:34:49.695 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:34:49.696 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:35:09.697 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:35:09.729 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:35:09.729 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:35:09.729 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:35:09.729 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:35:09.729 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:35:09.729 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:35:09.729 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:35:09.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:35:09.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:35:09.732 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:35:09.732 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:35:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:35:09.732 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:35:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:35:09.732 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:35:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:35:09.732 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:35:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:35:09.732 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:35:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:35:09.732 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:35:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:35:09.732 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:35:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:35:09.732 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:35:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:35:09.732 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:35:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:35:09.732 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:35:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:35:09.732 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:35:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:35:09.732 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:35:09.732 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:35:09.732 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:35:09.732 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:35:09.732 139328 ERROR oslo_service.periodic_task >2018-06-24 15:35:39.614 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:35:39.615 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:35:39.622 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:35:39.622 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:35:39.655 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:35:39.655 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:35:39.655 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:35:39.655 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:35:39.655 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:35:39.655 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:35:39.655 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:35:39.657 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:35:39.657 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:35:39.657 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:35:49.658 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:35:49.695 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:35:49.695 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:35:49.695 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:35:49.695 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:35:49.695 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:35:49.695 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:35:49.695 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:35:49.699 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:35:49.699 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:35:49.700 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:36:09.700 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:36:09.734 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:36:09.734 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:36:09.734 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:36:09.734 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:36:09.734 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:36:09.734 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:36:09.734 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:36:09.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:36:09.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:36:09.737 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:36:09.737 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:36:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:36:09.737 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:36:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:36:09.737 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:36:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:36:09.737 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:36:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:36:09.737 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:36:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:36:09.737 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:36:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:36:09.737 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:36:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:36:09.737 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:36:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:36:09.737 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:36:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:36:09.737 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:36:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:36:09.737 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:36:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:36:09.737 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:36:09.737 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:36:09.737 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:36:09.737 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:36:09.737 139328 ERROR oslo_service.periodic_task >2018-06-24 15:36:39.618 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:36:39.619 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:36:39.626 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:36:39.626 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:36:39.660 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:36:39.660 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:36:39.660 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:36:39.660 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:36:39.660 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:36:39.660 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:36:39.660 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:36:39.663 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:36:39.664 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:36:39.665 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:36:49.665 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:36:49.699 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:36:49.699 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:36:49.699 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:36:49.699 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:36:49.699 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:36:49.699 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:36:49.699 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:36:49.701 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:36:49.702 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:36:49.702 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:37:09.703 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:37:09.737 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:37:09.737 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:37:09.737 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:37:09.737 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:37:09.737 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:37:09.737 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:37:09.737 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:37:09.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:37:09.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:37:09.740 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:37:09.740 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:37:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:37:09.740 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:37:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:37:09.740 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:37:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:37:09.740 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:37:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:37:09.740 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:37:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:37:09.740 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:37:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:37:09.740 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:37:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:37:09.740 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:37:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:37:09.740 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:37:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:37:09.740 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:37:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:37:09.740 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:37:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:37:09.740 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:37:09.740 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:37:09.740 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:37:09.740 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:37:09.740 139328 ERROR oslo_service.periodic_task >2018-06-24 15:37:39.621 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:37:39.622 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:37:39.630 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:37:39.630 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:37:39.666 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:37:39.666 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:37:39.666 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:37:39.666 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:37:39.666 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:37:39.666 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:37:39.666 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:37:39.668 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:37:39.668 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:37:39.668 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:37:49.669 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:37:49.725 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:37:49.725 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:37:49.725 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:37:49.725 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:37:49.725 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:37:49.725 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:37:49.725 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:37:49.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:37:49.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.096 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:37:49.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:38:09.728 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:38:09.760 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:38:09.760 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:38:09.760 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:38:09.760 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:38:09.760 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:38:09.760 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:38:09.760 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:38:09.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:38:09.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.132 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:38:09.763 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:38:09.763 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:38:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:38:09.763 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:38:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:38:09.763 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:38:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:38:09.763 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:38:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:38:09.763 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:38:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:38:09.763 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:38:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:38:09.763 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:38:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:38:09.763 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:38:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:38:09.763 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:38:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:38:09.763 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:38:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:38:09.763 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:38:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:38:09.763 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:38:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:38:09.763 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:38:09.763 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:38:09.763 139328 ERROR oslo_service.periodic_task >2018-06-24 15:38:39.624 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:38:39.626 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:38:39.632 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:38:39.633 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:38:39.665 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:38:39.665 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:38:39.665 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:38:39.665 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:38:39.665 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:38:39.665 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:38:39.665 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:38:39.668 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:38:39.668 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:38:39.669 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:38:49.669 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:38:49.705 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:38:49.705 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:38:49.705 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:38:49.705 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:38:49.705 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:38:49.705 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:38:49.705 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:38:49.707 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:38:49.707 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:38:49.707 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:39:09.708 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:39:09.741 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:39:09.741 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:39:09.741 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:39:09.741 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:39:09.741 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:39:09.741 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:39:09.741 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:39:09.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:39:09.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:39:09.744 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:39:09.744 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:39:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:39:09.744 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:39:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:39:09.744 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:39:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:39:09.744 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:39:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:39:09.744 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:39:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:39:09.744 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:39:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:39:09.744 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:39:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:39:09.744 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:39:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:39:09.744 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:39:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:39:09.744 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:39:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:39:09.744 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:39:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:39:09.744 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:39:09.744 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:39:09.744 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:39:09.744 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:39:09.744 139328 ERROR oslo_service.periodic_task >2018-06-24 15:39:39.625 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:39:39.626 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:39:39.635 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:39:39.635 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:39:39.671 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:39:39.671 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:39:39.671 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:39:39.671 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:39:39.671 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:39:39.671 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:39:39.671 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:39:39.674 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:39:39.674 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:39:39.674 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:39:49.675 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:39:49.708 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:39:49.708 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:39:49.708 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:39:49.708 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:39:49.708 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:39:49.708 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:39:49.708 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:39:49.710 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:39:49.710 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:39:49.711 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:40:09.711 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:40:09.750 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:40:09.750 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:40:09.750 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:40:09.750 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:40:09.750 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:40:09.750 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:40:09.750 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:40:09.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:40:09.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:40:09.753 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:40:09.753 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:40:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:40:09.753 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:40:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:40:09.753 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:40:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:40:09.753 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:40:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:40:09.753 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:40:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:40:09.753 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:40:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:40:09.753 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:40:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:40:09.753 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:40:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:40:09.753 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:40:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:40:09.753 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:40:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:40:09.753 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:40:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:40:09.753 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:40:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:40:09.753 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:40:09.753 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:40:09.753 139328 ERROR oslo_service.periodic_task >2018-06-24 15:40:39.625 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:40:39.627 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:40:39.635 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:40:39.636 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:40:39.679 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:40:39.679 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:40:39.679 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:40:39.679 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:40:39.679 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:40:39.679 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:40:39.679 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:40:39.682 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:40:39.682 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.045 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:40:39.682 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:40:49.683 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:40:49.715 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:40:49.715 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:40:49.715 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:40:49.715 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:40:49.715 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:40:49.715 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:40:49.715 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:40:49.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:40:49.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.081 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:40:49.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:41:09.718 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:41:09.751 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:41:09.751 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:41:09.751 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:41:09.751 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:41:09.751 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:41:09.751 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:41:09.751 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:41:09.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:41:09.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:41:09.754 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:41:09.754 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:41:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:41:09.754 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:41:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:41:09.754 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:41:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:41:09.754 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:41:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:41:09.754 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:41:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:41:09.754 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:41:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:41:09.754 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:41:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:41:09.754 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:41:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:41:09.754 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:41:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:41:09.754 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:41:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:41:09.754 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:41:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:41:09.754 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:41:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:41:09.754 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:41:09.754 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:41:09.754 139328 ERROR oslo_service.periodic_task >2018-06-24 15:41:39.625 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:41:39.627 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:41:39.635 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:41:39.636 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:41:39.675 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:41:39.675 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:41:39.675 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:41:39.675 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:41:39.675 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:41:39.675 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:41:39.675 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:41:39.678 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:41:39.678 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.042 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:41:39.679 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:41:49.679 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:41:49.712 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:41:49.712 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:41:49.712 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:41:49.712 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:41:49.712 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:41:49.712 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:41:49.712 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:41:49.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:41:49.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:41:49.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:42:09.715 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:42:09.747 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:42:09.747 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:42:09.747 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:42:09.747 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:42:09.747 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:42:09.747 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:42:09.747 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:42:09.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:42:09.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:42:09.751 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:42:09.751 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:42:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:42:09.751 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:42:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:42:09.751 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:42:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:42:09.751 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:42:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:42:09.751 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:42:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:42:09.751 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:42:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:42:09.751 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:42:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:42:09.751 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:42:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:42:09.751 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:42:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:42:09.751 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:42:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:42:09.751 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:42:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:42:09.751 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:42:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:42:09.751 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:42:09.751 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:42:09.751 139328 ERROR oslo_service.periodic_task >2018-06-24 15:42:39.622 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:42:39.623 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:42:39.631 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:42:39.631 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:42:39.666 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:42:39.666 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:42:39.666 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:42:39.666 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:42:39.666 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:42:39.666 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:42:39.666 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:42:39.668 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:42:39.668 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:42:39.668 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:42:49.669 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:42:49.701 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:42:49.701 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:42:49.701 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:42:49.701 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:42:49.701 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:42:49.701 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:42:49.701 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:42:49.703 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:42:49.704 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:42:49.704 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:43:09.705 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:43:09.738 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:43:09.738 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:43:09.738 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:43:09.738 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:43:09.738 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:43:09.738 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:43:09.738 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:43:09.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:43:09.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:43:09.741 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:43:09.741 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:43:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:43:09.741 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:43:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:43:09.741 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:43:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:43:09.741 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:43:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:43:09.741 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:43:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:43:09.741 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:43:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:43:09.741 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:43:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:43:09.741 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:43:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:43:09.741 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:43:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:43:09.741 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:43:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:43:09.741 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:43:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:43:09.741 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:43:09.741 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:43:09.741 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:43:09.741 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:43:09.741 139328 ERROR oslo_service.periodic_task >2018-06-24 15:43:39.622 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:43:39.623 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:43:39.631 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:43:39.632 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:43:39.665 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:43:39.665 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:43:39.665 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:43:39.665 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:43:39.665 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:43:39.665 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:43:39.665 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:43:39.667 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:43:39.668 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:43:39.668 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:43:49.669 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:43:49.707 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:43:49.707 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:43:49.707 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:43:49.707 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:43:49.707 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:43:49.707 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:43:49.707 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:43:49.710 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:43:49.710 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:43:49.711 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:44:09.711 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:44:09.743 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:44:09.743 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:44:09.743 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:44:09.743 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:44:09.743 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:44:09.743 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:44:09.743 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:44:09.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:44:09.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:44:09.746 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:44:09.746 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:44:09.746 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:44:09.746 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:44:09.746 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:44:09.746 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:44:09.746 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:44:09.746 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:44:09.746 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:44:09.746 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:44:09.746 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:44:09.746 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:44:09.746 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:44:09.746 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:44:09.746 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:44:09.746 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:44:09.746 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:44:09.746 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:44:09.746 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:44:09.746 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:44:09.746 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:44:09.746 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:44:09.746 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:44:09.746 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:44:09.746 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:44:09.746 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:44:09.746 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:44:09.746 139328 ERROR oslo_service.periodic_task >2018-06-24 15:44:39.627 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:44:39.628 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:44:39.636 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:44:39.637 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:44:39.673 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:44:39.673 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:44:39.673 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:44:39.673 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:44:39.673 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:44:39.673 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:44:39.673 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:44:39.675 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:44:39.675 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:44:39.676 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:44:49.676 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:44:49.710 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:44:49.710 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:44:49.710 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:44:49.710 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:44:49.710 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:44:49.710 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:44:49.710 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:44:49.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:44:49.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:44:49.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:45:09.714 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:45:09.747 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:45:09.747 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:45:09.747 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:45:09.747 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:45:09.747 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:45:09.747 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:45:09.747 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:45:09.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:45:09.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:45:09.750 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:45:09.750 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:45:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:45:09.750 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:45:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:45:09.750 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:45:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:45:09.750 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:45:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:45:09.750 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:45:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:45:09.750 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:45:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:45:09.750 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:45:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:45:09.750 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:45:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:45:09.750 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:45:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:45:09.750 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:45:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:45:09.750 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:45:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:45:09.750 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:45:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:45:09.750 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:45:09.750 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:45:09.750 139328 ERROR oslo_service.periodic_task >2018-06-24 15:45:39.631 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:45:39.632 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:45:39.640 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:45:39.640 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:45:39.674 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:45:39.674 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:45:39.674 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:45:39.674 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:45:39.674 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:45:39.674 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:45:39.674 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:45:39.676 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:45:39.677 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:45:39.677 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:45:49.678 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:45:49.711 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:45:49.711 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:45:49.711 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:45:49.711 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:45:49.711 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:45:49.711 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:45:49.711 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:45:49.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:45:49.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:45:49.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:46:09.715 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:46:09.749 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:46:09.749 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:46:09.749 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:46:09.749 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:46:09.749 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:46:09.749 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:46:09.749 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:46:09.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:46:09.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:46:09.753 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:46:09.753 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:46:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:46:09.753 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:46:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:46:09.753 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:46:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:46:09.753 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:46:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:46:09.753 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:46:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:46:09.753 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:46:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:46:09.753 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:46:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:46:09.753 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:46:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:46:09.753 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:46:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:46:09.753 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:46:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:46:09.753 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:46:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:46:09.753 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:46:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:46:09.753 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:46:09.753 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:46:09.753 139328 ERROR oslo_service.periodic_task >2018-06-24 15:46:39.634 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:46:39.635 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:46:39.644 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:46:39.645 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:46:39.696 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:46:39.696 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:46:39.696 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:46:39.696 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:46:39.696 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:46:39.696 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:46:39.696 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:46:39.698 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:46:39.699 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.054 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:46:39.699 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:46:49.700 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:46:49.733 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:46:49.733 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:46:49.733 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:46:49.733 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:46:49.733 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:46:49.733 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:46:49.733 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:46:49.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:46:49.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.091 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:46:49.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:47:09.737 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:47:09.773 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:47:09.773 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:47:09.773 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:47:09.773 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:47:09.773 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:47:09.773 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:47:09.773 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:47:09.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:47:09.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.131 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:47:09.777 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:47:09.777 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:47:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:47:09.777 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:47:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:47:09.777 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:47:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:47:09.777 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:47:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:47:09.777 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:47:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:47:09.777 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:47:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:47:09.777 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:47:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:47:09.777 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:47:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:47:09.777 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:47:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:47:09.777 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:47:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:47:09.777 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:47:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:47:09.777 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:47:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:47:09.777 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:47:09.777 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:47:09.777 139328 ERROR oslo_service.periodic_task >2018-06-24 15:47:39.638 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:47:39.639 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:47:39.647 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:47:39.647 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:47:39.678 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:47:39.678 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:47:39.678 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:47:39.678 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:47:39.678 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:47:39.678 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:47:39.678 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:47:39.680 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:47:39.680 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.033 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:47:39.681 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:47:49.681 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:47:49.713 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:47:49.713 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:47:49.713 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:47:49.713 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:47:49.713 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:47:49.713 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:47:49.713 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:47:49.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:47:49.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.067 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:47:49.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:48:09.716 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:48:09.750 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:48:09.750 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:48:09.750 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:48:09.750 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:48:09.750 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:48:09.750 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:48:09.750 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:48:09.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:48:09.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.105 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:48:09.753 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:48:09.753 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:48:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:48:09.753 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:48:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:48:09.753 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:48:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:48:09.753 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:48:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:48:09.753 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:48:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:48:09.753 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:48:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:48:09.753 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:48:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:48:09.753 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:48:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:48:09.753 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:48:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:48:09.753 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:48:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:48:09.753 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:48:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:48:09.753 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:48:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:48:09.753 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:48:09.753 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:48:09.753 139328 ERROR oslo_service.periodic_task >2018-06-24 15:48:39.638 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:48:39.639 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:48:39.664 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:48:39.664 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:48:39.731 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:48:39.731 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:48:39.731 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:48:39.731 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:48:39.731 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:48:39.731 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:48:39.731 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:48:39.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:48:39.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:48:39.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:48:49.738 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:48:49.771 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:48:49.771 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:48:49.771 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:48:49.771 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:48:49.771 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:48:49.771 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:48:49.771 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:48:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:48:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:48:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:49:09.775 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:49:09.816 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:49:09.816 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:49:09.816 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:49:09.816 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:49:09.816 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:49:09.816 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:49:09.816 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:49:09.819 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:49:09.819 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.155 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:49:09.820 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:49:09.820 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:49:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:49:09.820 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:49:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:49:09.820 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:49:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:49:09.820 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:49:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:49:09.820 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:49:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:49:09.820 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:49:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:49:09.820 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:49:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:49:09.820 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:49:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:49:09.820 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:49:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:49:09.820 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:49:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:49:09.820 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:49:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:49:09.820 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:49:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:49:09.820 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:49:09.820 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:49:09.820 139328 ERROR oslo_service.periodic_task >2018-06-24 15:49:39.641 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:49:39.642 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:49:39.650 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:49:39.651 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:49:39.684 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:49:39.684 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:49:39.684 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:49:39.684 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:49:39.684 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:49:39.684 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:49:39.684 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:49:39.686 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:49:39.686 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:49:39.687 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:49:49.687 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:49:49.722 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:49:49.722 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:49:49.722 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:49:49.722 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:49:49.722 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:49:49.722 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:49:49.722 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:49:49.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:49:49.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:49:49.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:50:09.725 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:50:09.766 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:50:09.766 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:50:09.766 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:50:09.766 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:50:09.766 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:50:09.766 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:50:09.766 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:50:09.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:50:09.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:50:09.769 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:50:09.769 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:50:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:50:09.769 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:50:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:50:09.769 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:50:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:50:09.769 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:50:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:50:09.769 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:50:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:50:09.769 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:50:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:50:09.769 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:50:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:50:09.769 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:50:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:50:09.769 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:50:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:50:09.769 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:50:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:50:09.769 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:50:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:50:09.769 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:50:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:50:09.769 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:50:09.769 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:50:09.769 139328 ERROR oslo_service.periodic_task >2018-06-24 15:50:39.640 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:50:39.641 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:50:39.647 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:50:39.647 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:50:39.679 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:50:39.679 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:50:39.679 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:50:39.679 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:50:39.679 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:50:39.679 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:50:39.679 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:50:39.681 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:50:39.682 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:50:39.682 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:50:49.683 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:50:49.714 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:50:49.714 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:50:49.714 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:50:49.714 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:50:49.714 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:50:49.714 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:50:49.714 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:50:49.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:50:49.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.068 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:50:49.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:51:09.717 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:51:09.752 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:51:09.752 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:51:09.752 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:51:09.752 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:51:09.752 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:51:09.752 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:51:09.752 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:51:09.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:51:09.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.107 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:51:09.755 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:51:09.755 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:51:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:51:09.755 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:51:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:51:09.755 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:51:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:51:09.755 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:51:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:51:09.755 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:51:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:51:09.755 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:51:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:51:09.755 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:51:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:51:09.755 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:51:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:51:09.755 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:51:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:51:09.755 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:51:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:51:09.755 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:51:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:51:09.755 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:51:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:51:09.755 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:51:09.755 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:51:09.755 139328 ERROR oslo_service.periodic_task >2018-06-24 15:51:39.636 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:51:39.637 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:51:39.644 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:51:39.644 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:51:39.679 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:51:39.679 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:51:39.679 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:51:39.679 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:51:39.679 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:51:39.679 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:51:39.679 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:51:39.681 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:51:39.681 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:51:39.681 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:51:49.682 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:51:49.715 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:51:49.715 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:51:49.715 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:51:49.715 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:51:49.715 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:51:49.715 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:51:49.715 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:51:49.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:51:49.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:51:49.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:52:09.719 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:52:09.751 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:52:09.751 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:52:09.751 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:52:09.751 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:52:09.751 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:52:09.751 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:52:09.751 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:52:09.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:52:09.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:52:09.754 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:52:09.754 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:52:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:52:09.754 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:52:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:52:09.754 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:52:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:52:09.754 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:52:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:52:09.754 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:52:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:52:09.754 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:52:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:52:09.754 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:52:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:52:09.754 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:52:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:52:09.754 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:52:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:52:09.754 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:52:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:52:09.754 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:52:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:52:09.754 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:52:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:52:09.754 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:52:09.754 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:52:09.754 139328 ERROR oslo_service.periodic_task >2018-06-24 15:52:39.635 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:52:39.635 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:52:39.642 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:52:39.643 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:52:39.677 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:52:39.677 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:52:39.677 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:52:39.677 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:52:39.677 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:52:39.677 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:52:39.677 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:52:39.679 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:52:39.679 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:52:39.680 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:52:49.680 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:52:49.721 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:52:49.721 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:52:49.721 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:52:49.721 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:52:49.721 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:52:49.721 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:52:49.721 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:52:49.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:52:49.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.081 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:52:49.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:53:09.725 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:53:09.756 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:53:09.756 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:53:09.756 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:53:09.756 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:53:09.756 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:53:09.756 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:53:09.756 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:53:09.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:53:09.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:53:09.759 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:53:09.759 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:53:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:53:09.759 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:53:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:53:09.759 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:53:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:53:09.759 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:53:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:53:09.759 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:53:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:53:09.759 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:53:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:53:09.759 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:53:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:53:09.759 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:53:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:53:09.759 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:53:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:53:09.759 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:53:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:53:09.759 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:53:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:53:09.759 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:53:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:53:09.759 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:53:09.759 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:53:09.759 139328 ERROR oslo_service.periodic_task >2018-06-24 15:53:39.630 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:53:39.632 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:53:39.639 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:53:39.640 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:53:39.673 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:53:39.673 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:53:39.673 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:53:39.673 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:53:39.673 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:53:39.673 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:53:39.673 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:53:39.675 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:53:39.675 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:53:39.675 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:53:49.676 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:53:49.709 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:53:49.709 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:53:49.709 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:53:49.709 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:53:49.709 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:53:49.709 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:53:49.709 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:53:49.711 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:53:49.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:53:49.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:54:09.712 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:54:09.747 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:54:09.747 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:54:09.747 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:54:09.747 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:54:09.747 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:54:09.747 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:54:09.747 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:54:09.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:54:09.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:54:09.750 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:54:09.750 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:54:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:54:09.750 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:54:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:54:09.750 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:54:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:54:09.750 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:54:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:54:09.750 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:54:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:54:09.750 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:54:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:54:09.750 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:54:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:54:09.750 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:54:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:54:09.750 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:54:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:54:09.750 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:54:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:54:09.750 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:54:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:54:09.750 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:54:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:54:09.750 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:54:09.750 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:54:09.750 139328 ERROR oslo_service.periodic_task >2018-06-24 15:54:39.631 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:54:39.632 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:54:39.638 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:54:39.639 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:54:39.679 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:54:39.679 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:54:39.679 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:54:39.679 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:54:39.679 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:54:39.679 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:54:39.679 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:54:39.681 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:54:39.681 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.042 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:54:39.681 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:54:49.682 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:54:49.716 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:54:49.716 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:54:49.716 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:54:49.716 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:54:49.716 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:54:49.716 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:54:49.716 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:54:49.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:54:49.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.08 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:54:49.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:55:09.720 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:55:09.754 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:55:09.754 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:55:09.754 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:55:09.754 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:55:09.754 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:55:09.754 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:55:09.754 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:55:09.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:55:09.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.118 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:55:09.757 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:55:09.757 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:55:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:55:09.757 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:55:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:55:09.757 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:55:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:55:09.757 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:55:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:55:09.757 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:55:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:55:09.757 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:55:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:55:09.757 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:55:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:55:09.757 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:55:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:55:09.757 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:55:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:55:09.757 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:55:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:55:09.757 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:55:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:55:09.757 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:55:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:55:09.757 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:55:09.757 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:55:09.757 139328 ERROR oslo_service.periodic_task >2018-06-24 15:55:39.628 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:55:39.629 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:55:39.636 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:55:39.636 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:55:39.669 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:55:39.669 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:55:39.669 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:55:39.669 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:55:39.669 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:55:39.669 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:55:39.669 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:55:39.671 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:55:39.671 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:55:39.672 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:55:49.672 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:55:49.706 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:55:49.706 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:55:49.706 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:55:49.706 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:55:49.706 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:55:49.706 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:55:49.706 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:55:49.708 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:55:49.709 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:55:49.709 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:56:09.709 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:56:09.743 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:56:09.743 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:56:09.743 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:56:09.743 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:56:09.743 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:56:09.743 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:56:09.743 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:56:09.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:56:09.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:56:09.745 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:56:09.745 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:56:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:56:09.745 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:56:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:56:09.745 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:56:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:56:09.745 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:56:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:56:09.745 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:56:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:56:09.745 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:56:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:56:09.745 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:56:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:56:09.745 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:56:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:56:09.745 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:56:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:56:09.745 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:56:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:56:09.745 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:56:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:56:09.745 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:56:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:56:09.745 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:56:09.745 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:56:09.745 139328 ERROR oslo_service.periodic_task >2018-06-24 15:56:39.627 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:56:39.628 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:56:39.635 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:56:39.636 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:56:39.669 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:56:39.669 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:56:39.669 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:56:39.669 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:56:39.669 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:56:39.669 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:56:39.669 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:56:39.672 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:56:39.672 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:56:39.672 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:56:49.673 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:56:49.708 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:56:49.708 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:56:49.708 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:56:49.708 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:56:49.708 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:56:49.708 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:56:49.708 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:56:49.710 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:56:49.710 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:56:49.711 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:57:09.711 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:57:09.744 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:57:09.744 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:57:09.744 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:57:09.744 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:57:09.744 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:57:09.744 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:57:09.744 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:57:09.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:57:09.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:57:09.747 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:57:09.747 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:57:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:57:09.747 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:57:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:57:09.747 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:57:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:57:09.747 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:57:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:57:09.747 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:57:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:57:09.747 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:57:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:57:09.747 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:57:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:57:09.747 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:57:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:57:09.747 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:57:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:57:09.747 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:57:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:57:09.747 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:57:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:57:09.747 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:57:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:57:09.747 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:57:09.747 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:57:09.747 139328 ERROR oslo_service.periodic_task >2018-06-24 15:57:39.628 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:57:39.629 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:57:39.637 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:57:39.637 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:57:39.675 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:57:39.675 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:57:39.675 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:57:39.675 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:57:39.675 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:57:39.675 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:57:39.675 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:57:39.677 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:57:39.677 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:57:39.678 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:57:49.678 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:57:49.712 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:57:49.712 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:57:49.712 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:57:49.712 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:57:49.712 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:57:49.712 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:57:49.712 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:57:49.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:57:49.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:57:49.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:58:09.715 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:58:09.768 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:58:09.768 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:58:09.768 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:58:09.768 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:58:09.768 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:58:09.768 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:58:09.768 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:58:09.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:58:09.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.133 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:58:09.771 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:58:09.771 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:58:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:58:09.771 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:58:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:58:09.771 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:58:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:58:09.771 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:58:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:58:09.771 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:58:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:58:09.771 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:58:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:58:09.771 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:58:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:58:09.771 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:58:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:58:09.771 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:58:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:58:09.771 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:58:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:58:09.771 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:58:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:58:09.771 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:58:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:58:09.771 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:58:09.771 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:58:09.771 139328 ERROR oslo_service.periodic_task >2018-06-24 15:58:39.633 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:58:39.634 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:58:39.642 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:58:39.643 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:58:39.679 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:58:39.679 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:58:39.679 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:58:39.679 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:58:39.679 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:58:39.679 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:58:39.679 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:58:39.681 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:58:39.682 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:58:39.682 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:58:49.683 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:58:49.718 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:58:49.718 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:58:49.718 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:58:49.718 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:58:49.718 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:58:49.718 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:58:49.718 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:58:49.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:58:49.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:58:49.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:59:09.721 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:59:09.755 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:59:09.755 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:59:09.755 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:59:09.755 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:59:09.755 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:59:09.755 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:59:09.755 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:59:09.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:59:09.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:59:09.758 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:59:09.758 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 15:59:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 15:59:09.758 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 15:59:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 15:59:09.758 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 15:59:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 15:59:09.758 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 15:59:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 15:59:09.758 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 15:59:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 15:59:09.758 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 15:59:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 15:59:09.758 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 15:59:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 15:59:09.758 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 15:59:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 15:59:09.758 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 15:59:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 15:59:09.758 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 15:59:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 15:59:09.758 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 15:59:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 15:59:09.758 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 15:59:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 15:59:09.758 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 15:59:09.758 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 15:59:09.758 139328 ERROR oslo_service.periodic_task >2018-06-24 15:59:39.629 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:59:39.630 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 15:59:39.636 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 15:59:39.637 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:59:39.671 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:59:39.671 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:59:39.671 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:59:39.671 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:59:39.671 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:59:39.671 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:59:39.671 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:59:39.673 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:59:39.674 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:59:39.674 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 15:59:49.675 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 15:59:49.710 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 15:59:49.710 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 15:59:49.710 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 15:59:49.710 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 15:59:49.710 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 15:59:49.710 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 15:59:49.710 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 15:59:49.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 15:59:49.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 15:59:49.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:00:09.713 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:00:09.747 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:00:09.747 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:00:09.747 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:00:09.747 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:00:09.747 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:00:09.747 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:00:09.747 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:00:09.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:00:09.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:00:09.750 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:00:09.750 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:00:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:00:09.750 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:00:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:00:09.750 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:00:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:00:09.750 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:00:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:00:09.750 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:00:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:00:09.750 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:00:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:00:09.750 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:00:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:00:09.750 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:00:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:00:09.750 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:00:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:00:09.750 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:00:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:00:09.750 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:00:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:00:09.750 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:00:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:00:09.750 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:00:09.750 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:00:09.750 139328 ERROR oslo_service.periodic_task >2018-06-24 16:00:39.631 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:00:39.633 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:00:39.641 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:00:39.642 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:00:39.675 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:00:39.675 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:00:39.675 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:00:39.675 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:00:39.675 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:00:39.675 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:00:39.675 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:00:39.680 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:00:39.680 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:00:39.680 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:00:49.681 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:00:49.714 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:00:49.714 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:00:49.714 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:00:49.714 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:00:49.714 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:00:49.714 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:00:49.714 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:00:49.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:00:49.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:00:49.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:01:09.719 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:01:09.754 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:01:09.754 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:01:09.754 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:01:09.754 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:01:09.754 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:01:09.754 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:01:09.754 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:01:09.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:01:09.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:01:09.758 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:01:09.758 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:01:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:01:09.758 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:01:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:01:09.758 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:01:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:01:09.758 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:01:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:01:09.758 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:01:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:01:09.758 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:01:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:01:09.758 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:01:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:01:09.758 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:01:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:01:09.758 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:01:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:01:09.758 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:01:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:01:09.758 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:01:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:01:09.758 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:01:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:01:09.758 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:01:09.758 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:01:09.758 139328 ERROR oslo_service.periodic_task >2018-06-24 16:01:39.629 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:01:39.630 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:01:39.637 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:01:39.638 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:01:39.675 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:01:39.675 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:01:39.675 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:01:39.675 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:01:39.675 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:01:39.675 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:01:39.675 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:01:39.678 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:01:39.678 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:01:39.678 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:01:49.679 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:01:49.712 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:01:49.712 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:01:49.712 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:01:49.712 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:01:49.712 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:01:49.712 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:01:49.712 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:01:49.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:01:49.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:01:49.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:02:09.716 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:02:09.748 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:02:09.748 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:02:09.748 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:02:09.748 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:02:09.748 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:02:09.748 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:02:09.748 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:02:09.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:02:09.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:02:09.750 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:02:09.750 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:02:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:02:09.750 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:02:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:02:09.750 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:02:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:02:09.750 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:02:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:02:09.750 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:02:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:02:09.750 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:02:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:02:09.750 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:02:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:02:09.750 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:02:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:02:09.750 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:02:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:02:09.750 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:02:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:02:09.750 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:02:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:02:09.750 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:02:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:02:09.750 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:02:09.750 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:02:09.750 139328 ERROR oslo_service.periodic_task >2018-06-24 16:02:39.632 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:02:39.633 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:02:39.640 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:02:39.642 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:02:39.678 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:02:39.678 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:02:39.678 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:02:39.678 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:02:39.678 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:02:39.678 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:02:39.678 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:02:39.680 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:02:39.680 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:02:39.680 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:02:49.681 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:02:49.715 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:02:49.715 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:02:49.715 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:02:49.715 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:02:49.715 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:02:49.715 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:02:49.715 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:02:49.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:02:49.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:02:49.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:03:09.719 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:03:09.770 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:03:09.770 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:03:09.770 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:03:09.770 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:03:09.770 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:03:09.770 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:03:09.770 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:03:09.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:03:09.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.13 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:03:09.773 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:03:09.773 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:03:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:03:09.773 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:03:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:03:09.773 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:03:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:03:09.773 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:03:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:03:09.773 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:03:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:03:09.773 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:03:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:03:09.773 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:03:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:03:09.773 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:03:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:03:09.773 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:03:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:03:09.773 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:03:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:03:09.773 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:03:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:03:09.773 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:03:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:03:09.773 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:03:09.773 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:03:09.773 139328 ERROR oslo_service.periodic_task >2018-06-24 16:03:39.634 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:03:39.635 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:03:39.671 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:03:39.671 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:03:39.671 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:03:39.671 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:03:39.671 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:03:39.671 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:03:39.671 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:03:39.673 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:03:39.673 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:03:39.674 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:03:49.674 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:03:49.709 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:03:49.709 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:03:49.709 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:03:49.709 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:03:49.709 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:03:49.709 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:03:49.709 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:03:49.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:03:49.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:03:49.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:04:09.713 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:04:09.749 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:04:09.749 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:04:09.749 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:04:09.749 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:04:09.749 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:04:09.749 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:04:09.749 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:04:09.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:04:09.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:04:09.752 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:04:09.752 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:04:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:04:09.752 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:04:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:04:09.752 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:04:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:04:09.752 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:04:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:04:09.752 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:04:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:04:09.752 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:04:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:04:09.752 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:04:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:04:09.752 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:04:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:04:09.752 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:04:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:04:09.752 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:04:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:04:09.752 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:04:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:04:09.752 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:04:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:04:09.752 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:04:09.752 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:04:09.752 139328 ERROR oslo_service.periodic_task >2018-06-24 16:04:39.634 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:04:39.635 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:04:39.641 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:04:39.642 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:04:39.674 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:04:39.674 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:04:39.674 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:04:39.674 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:04:39.674 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:04:39.674 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:04:39.674 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:04:39.676 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:04:39.677 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:04:39.677 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:04:49.677 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:04:49.710 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:04:49.710 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:04:49.710 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:04:49.710 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:04:49.710 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:04:49.710 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:04:49.710 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:04:49.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:04:49.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:04:49.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:05:09.713 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:05:09.747 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:05:09.747 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:05:09.747 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:05:09.747 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:05:09.747 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:05:09.747 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:05:09.747 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:05:09.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:05:09.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.107 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:05:09.749 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:05:09.749 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:05:09.749 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:05:09.749 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:05:09.749 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:05:09.749 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:05:09.749 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:05:09.749 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:05:09.749 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:05:09.749 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:05:09.749 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:05:09.749 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:05:09.749 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:05:09.749 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:05:09.749 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:05:09.749 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:05:09.749 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:05:09.749 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:05:09.749 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:05:09.749 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:05:09.749 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:05:09.749 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:05:09.749 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:05:09.749 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:05:09.749 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:05:09.749 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:05:09.749 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:05:09.749 139328 ERROR oslo_service.periodic_task >2018-06-24 16:05:39.631 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:05:39.632 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:05:39.641 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:05:39.642 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:05:39.692 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:05:39.692 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:05:39.692 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:05:39.692 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:05:39.692 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:05:39.692 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:05:39.692 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:05:39.694 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:05:39.694 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.052 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:05:39.695 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:05:49.695 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:05:49.730 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:05:49.730 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:05:49.730 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:05:49.730 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:05:49.730 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:05:49.730 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:05:49.730 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:05:49.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:05:49.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.091 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:05:49.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:06:09.734 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:06:09.770 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:06:09.770 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:06:09.770 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:06:09.770 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:06:09.770 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:06:09.770 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:06:09.770 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:06:09.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:06:09.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.13 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:06:09.772 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:06:09.772 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:06:09.772 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:06:09.772 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:06:09.772 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:06:09.772 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:06:09.772 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:06:09.772 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:06:09.772 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:06:09.772 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:06:09.772 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:06:09.772 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:06:09.772 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:06:09.772 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:06:09.772 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:06:09.772 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:06:09.772 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:06:09.772 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:06:09.772 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:06:09.772 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:06:09.772 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:06:09.772 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:06:09.772 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:06:09.772 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:06:09.772 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:06:09.772 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:06:09.772 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:06:09.772 139328 ERROR oslo_service.periodic_task >2018-06-24 16:06:39.634 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:06:39.635 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:06:39.655 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:06:39.656 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:06:39.697 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:06:39.697 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:06:39.697 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:06:39.697 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:06:39.697 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:06:39.697 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:06:39.697 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:06:39.699 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:06:39.699 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.044 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:06:39.700 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:06:49.700 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:06:49.734 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:06:49.734 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:06:49.734 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:06:49.734 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:06:49.734 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:06:49.734 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:06:49.734 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:06:49.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:06:49.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.08 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:06:49.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:07:09.737 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:07:09.772 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:07:09.772 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:07:09.772 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:07:09.772 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:07:09.772 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:07:09.772 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:07:09.772 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:07:09.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:07:09.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.119 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:07:09.775 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:07:09.775 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:07:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:07:09.775 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:07:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:07:09.775 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:07:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:07:09.775 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:07:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:07:09.775 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:07:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:07:09.775 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:07:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:07:09.775 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:07:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:07:09.775 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:07:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:07:09.775 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:07:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:07:09.775 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:07:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:07:09.775 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:07:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:07:09.775 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:07:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:07:09.775 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:07:09.775 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:07:09.775 139328 ERROR oslo_service.periodic_task >2018-06-24 16:07:39.637 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:07:39.638 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:07:39.645 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:07:39.646 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:07:39.683 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:07:39.683 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:07:39.683 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:07:39.683 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:07:39.683 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:07:39.683 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:07:39.683 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:07:39.685 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:07:39.686 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:07:39.686 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:07:49.687 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:07:49.720 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:07:49.720 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:07:49.720 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:07:49.720 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:07:49.720 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:07:49.720 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:07:49.720 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:07:49.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:07:49.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:07:49.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:08:09.723 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:08:09.758 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:08:09.758 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:08:09.758 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:08:09.758 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:08:09.758 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:08:09.758 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:08:09.758 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:08:09.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:08:09.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:08:09.761 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:08:09.761 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:08:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:08:09.761 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:08:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:08:09.761 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:08:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:08:09.761 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:08:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:08:09.761 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:08:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:08:09.761 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:08:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:08:09.761 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:08:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:08:09.761 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:08:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:08:09.761 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:08:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:08:09.761 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:08:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:08:09.761 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:08:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:08:09.761 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:08:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:08:09.761 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:08:09.761 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:08:09.761 139328 ERROR oslo_service.periodic_task >2018-06-24 16:08:39.633 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:08:39.634 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:09:39.631 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:09:39.632 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:09:39.638 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:09:39.639 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:09:39.674 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:09:39.674 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:09:39.674 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:09:39.674 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:09:39.674 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:09:39.674 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:09:39.674 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:09:39.677 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:09:39.677 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:09:39.677 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:09:49.678 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:09:49.714 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:09:49.714 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:09:49.714 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:09:49.714 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:09:49.714 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:09:49.714 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:09:49.714 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:09:49.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:09:49.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:09:49.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:10:09.717 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:10:09.751 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:10:09.751 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:10:09.751 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:10:09.751 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:10:09.751 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:10:09.751 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:10:09.751 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:10:09.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:10:09.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:10:09.754 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:10:09.754 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:10:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:10:09.754 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:10:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:10:09.754 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:10:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:10:09.754 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:10:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:10:09.754 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:10:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:10:09.754 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:10:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:10:09.754 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:10:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:10:09.754 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:10:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:10:09.754 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:10:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:10:09.754 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:10:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:10:09.754 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:10:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:10:09.754 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:10:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:10:09.754 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:10:09.754 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:10:09.754 139328 ERROR oslo_service.periodic_task >2018-06-24 16:10:39.635 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:10:39.636 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:10:39.643 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:10:39.643 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:10:39.677 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:10:39.677 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:10:39.677 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:10:39.677 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:10:39.677 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:10:39.677 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:10:39.677 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:10:39.679 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:10:39.680 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:10:39.680 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:10:49.681 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:10:49.715 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:10:49.715 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:10:49.715 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:10:49.715 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:10:49.715 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:10:49.715 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:10:49.715 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:10:49.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:10:49.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:10:49.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:11:09.718 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:11:09.751 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:11:09.751 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:11:09.751 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:11:09.751 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:11:09.751 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:11:09.751 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:11:09.751 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:11:09.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:11:09.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:11:09.754 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:11:09.754 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:11:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:11:09.754 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:11:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:11:09.754 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:11:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:11:09.754 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:11:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:11:09.754 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:11:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:11:09.754 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:11:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:11:09.754 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:11:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:11:09.754 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:11:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:11:09.754 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:11:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:11:09.754 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:11:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:11:09.754 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:11:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:11:09.754 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:11:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:11:09.754 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:11:09.754 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:11:09.754 139328 ERROR oslo_service.periodic_task >2018-06-24 16:11:39.636 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:11:39.637 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:11:39.649 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:11:39.649 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:11:39.682 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:11:39.682 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:11:39.682 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:11:39.682 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:11:39.682 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:11:39.682 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:11:39.682 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:11:39.684 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:11:39.684 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:11:39.684 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:11:49.685 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:11:49.718 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:11:49.718 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:11:49.718 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:11:49.718 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:11:49.718 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:11:49.718 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:11:49.718 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:11:49.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:11:49.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:11:49.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:12:09.722 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:12:09.754 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:12:09.754 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:12:09.754 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:12:09.754 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:12:09.754 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:12:09.754 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:12:09.754 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:12:09.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:12:09.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.107 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:12:09.757 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:12:09.757 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:12:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:12:09.757 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:12:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:12:09.757 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:12:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:12:09.757 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:12:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:12:09.757 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:12:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:12:09.757 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:12:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:12:09.757 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:12:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:12:09.757 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:12:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:12:09.757 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:12:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:12:09.757 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:12:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:12:09.757 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:12:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:12:09.757 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:12:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:12:09.757 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:12:09.757 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:12:09.757 139328 ERROR oslo_service.periodic_task >2018-06-24 16:12:39.639 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:12:39.640 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:12:39.647 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:12:39.648 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:12:39.682 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:12:39.682 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:12:39.682 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:12:39.682 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:12:39.682 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:12:39.682 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:12:39.682 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:12:39.684 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:12:39.684 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:12:39.684 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:12:49.685 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:12:49.719 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:12:49.719 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:12:49.719 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:12:49.719 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:12:49.719 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:12:49.719 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:12:49.719 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:12:49.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:12:49.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:12:49.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:13:09.722 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:13:09.756 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:13:09.756 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:13:09.756 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:13:09.756 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:13:09.756 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:13:09.756 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:13:09.756 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:13:09.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:13:09.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:13:09.759 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:13:09.759 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:13:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:13:09.759 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:13:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:13:09.759 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:13:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:13:09.759 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:13:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:13:09.759 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:13:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:13:09.759 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:13:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:13:09.759 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:13:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:13:09.759 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:13:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:13:09.759 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:13:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:13:09.759 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:13:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:13:09.759 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:13:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:13:09.759 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:13:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:13:09.759 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:13:09.759 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:13:09.759 139328 ERROR oslo_service.periodic_task >2018-06-24 16:13:39.640 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:13:39.642 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:13:39.649 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:13:39.650 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:13:39.687 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:13:39.687 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:13:39.687 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:13:39.687 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:13:39.687 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:13:39.687 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:13:39.687 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:13:39.690 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:13:39.691 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.041 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:13:39.691 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:13:49.692 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:13:49.728 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:13:49.728 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:13:49.728 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:13:49.728 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:13:49.728 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:13:49.728 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:13:49.728 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:13:49.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:13:49.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.081 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:13:49.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:14:09.732 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:14:09.768 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:14:09.768 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:14:09.768 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:14:09.768 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:14:09.768 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:14:09.768 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:14:09.768 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:14:09.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:14:09.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.121 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:14:09.771 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:14:09.771 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:14:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:14:09.771 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:14:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:14:09.771 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:14:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:14:09.771 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:14:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:14:09.771 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:14:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:14:09.771 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:14:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:14:09.771 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:14:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:14:09.771 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:14:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:14:09.771 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:14:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:14:09.771 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:14:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:14:09.771 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:14:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:14:09.771 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:14:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:14:09.771 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:14:09.771 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:14:09.771 139328 ERROR oslo_service.periodic_task >2018-06-24 16:14:39.643 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:14:39.643 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:14:39.652 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:14:39.653 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:14:39.690 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:14:39.690 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:14:39.690 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:14:39.690 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:14:39.690 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:14:39.690 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:14:39.690 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:14:39.693 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:14:39.693 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:14:39.693 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:14:49.694 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:14:49.727 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:14:49.727 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:14:49.727 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:14:49.727 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:14:49.727 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:14:49.727 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:14:49.727 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:14:49.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:14:49.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:14:49.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:15:09.731 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:15:09.767 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:15:09.767 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:15:09.767 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:15:09.767 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:15:09.767 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:15:09.767 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:15:09.767 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:15:09.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:15:09.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:15:09.771 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:15:09.771 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:15:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:15:09.771 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:15:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:15:09.771 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:15:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:15:09.771 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:15:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:15:09.771 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:15:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:15:09.771 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:15:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:15:09.771 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:15:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:15:09.771 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:15:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:15:09.771 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:15:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:15:09.771 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:15:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:15:09.771 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:15:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:15:09.771 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:15:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:15:09.771 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:15:09.771 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:15:09.771 139328 ERROR oslo_service.periodic_task >2018-06-24 16:15:39.643 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:15:39.644 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:15:39.653 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:15:39.654 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:15:39.695 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:15:39.695 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:15:39.695 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:15:39.695 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:15:39.695 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:15:39.695 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:15:39.695 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:15:39.697 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:15:39.698 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.043 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:15:39.698 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:15:49.699 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:15:49.732 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:15:49.732 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:15:49.732 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:15:49.732 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:15:49.732 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:15:49.732 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:15:49.732 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:15:49.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:15:49.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:15:49.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:16:09.735 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:16:09.769 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:16:09.769 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:16:09.769 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:16:09.769 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:16:09.769 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:16:09.769 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:16:09.769 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:16:09.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:16:09.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:16:09.772 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:16:09.772 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:16:09.772 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:16:09.772 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:16:09.772 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:16:09.772 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:16:09.772 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:16:09.772 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:16:09.772 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:16:09.772 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:16:09.772 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:16:09.772 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:16:09.772 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:16:09.772 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:16:09.772 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:16:09.772 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:16:09.772 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:16:09.772 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:16:09.772 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:16:09.772 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:16:09.772 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:16:09.772 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:16:09.772 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:16:09.772 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:16:09.772 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:16:09.772 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:16:09.772 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:16:09.772 139328 ERROR oslo_service.periodic_task >2018-06-24 16:16:39.643 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:16:39.644 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:16:39.654 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:16:39.655 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:16:39.694 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:16:39.694 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:16:39.694 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:16:39.694 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:16:39.694 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:16:39.694 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:16:39.694 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:16:39.697 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:16:39.697 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.041 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:16:39.697 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:16:49.698 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:16:49.731 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:16:49.731 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:16:49.731 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:16:49.731 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:16:49.731 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:16:49.731 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:16:49.731 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:16:49.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:16:49.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:16:49.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:17:09.734 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:17:09.768 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:17:09.768 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:17:09.768 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:17:09.768 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:17:09.768 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:17:09.768 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:17:09.768 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:17:09.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:17:09.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:17:09.771 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:17:09.771 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:17:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:17:09.771 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:17:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:17:09.771 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:17:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:17:09.771 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:17:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:17:09.771 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:17:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:17:09.771 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:17:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:17:09.771 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:17:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:17:09.771 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:17:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:17:09.771 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:17:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:17:09.771 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:17:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:17:09.771 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:17:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:17:09.771 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:17:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:17:09.771 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:17:09.771 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:17:09.771 139328 ERROR oslo_service.periodic_task >2018-06-24 16:17:39.642 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:17:39.643 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:17:39.657 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:17:39.658 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:17:39.691 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:17:39.691 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:17:39.691 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:17:39.691 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:17:39.691 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:17:39.691 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:17:39.691 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:17:39.693 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:17:39.693 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:17:39.694 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:17:49.694 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:17:49.727 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:17:49.727 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:17:49.727 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:17:49.727 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:17:49.727 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:17:49.727 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:17:49.727 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:17:49.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:17:49.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:17:49.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:18:09.731 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:18:09.765 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:18:09.765 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:18:09.765 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:18:09.765 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:18:09.765 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:18:09.765 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:18:09.765 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:18:09.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:18:09.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:18:09.768 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:18:09.768 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:18:09.768 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:18:09.768 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:18:09.768 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:18:09.768 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:18:09.768 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:18:09.768 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:18:09.768 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:18:09.768 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:18:09.768 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:18:09.768 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:18:09.768 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:18:09.768 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:18:09.768 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:18:09.768 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:18:09.768 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:18:09.768 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:18:09.768 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:18:09.768 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:18:09.768 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:18:09.768 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:18:09.768 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:18:09.768 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:18:09.768 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:18:09.768 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:18:09.768 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:18:09.768 139328 ERROR oslo_service.periodic_task >2018-06-24 16:18:39.639 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:18:39.641 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:18:39.648 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:18:39.648 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:18:39.697 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:18:39.697 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:18:39.697 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:18:39.697 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:18:39.697 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:18:39.697 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:18:39.697 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:18:39.700 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:18:39.700 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.051 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:18:39.700 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:18:49.701 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:18:49.733 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:18:49.733 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:18:49.733 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:18:49.733 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:18:49.733 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:18:49.733 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:18:49.733 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:18:49.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:18:49.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.087 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:18:49.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:19:09.736 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:19:09.771 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:19:09.771 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:19:09.771 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:19:09.771 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:19:09.771 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:19:09.771 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:19:09.771 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:19:09.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:19:09.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.124 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:19:09.774 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:19:09.774 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:19:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:19:09.774 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:19:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:19:09.774 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:19:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:19:09.774 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:19:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:19:09.774 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:19:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:19:09.774 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:19:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:19:09.774 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:19:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:19:09.774 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:19:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:19:09.774 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:19:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:19:09.774 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:19:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:19:09.774 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:19:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:19:09.774 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:19:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:19:09.774 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:19:09.774 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:19:09.774 139328 ERROR oslo_service.periodic_task >2018-06-24 16:19:39.635 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:19:39.636 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:19:39.644 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:19:39.645 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:19:39.682 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:19:39.682 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:19:39.682 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:19:39.682 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:19:39.682 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:19:39.682 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:19:39.682 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:19:39.684 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:19:39.684 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:19:39.685 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:19:49.685 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:19:49.726 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:19:49.726 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:19:49.726 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:19:49.726 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:19:49.726 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:19:49.726 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:19:49.726 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:19:49.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:19:49.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.084 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:19:49.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:20:09.730 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:20:09.763 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:20:09.763 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:20:09.763 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:20:09.763 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:20:09.763 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:20:09.763 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:20:09.763 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:20:09.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:20:09.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.122 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:20:09.767 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:20:09.767 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:20:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:20:09.767 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:20:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:20:09.767 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:20:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:20:09.767 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:20:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:20:09.767 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:20:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:20:09.767 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:20:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:20:09.767 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:20:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:20:09.767 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:20:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:20:09.767 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:20:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:20:09.767 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:20:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:20:09.767 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:20:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:20:09.767 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:20:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:20:09.767 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:20:09.767 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:20:09.767 139328 ERROR oslo_service.periodic_task >2018-06-24 16:20:39.639 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:20:39.640 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:20:39.646 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:20:39.647 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:20:39.679 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:20:39.679 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:20:39.679 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:20:39.679 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:20:39.679 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:20:39.679 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:20:39.679 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:20:39.680 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:20:39.681 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:20:39.681 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:20:49.681 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:20:49.717 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:20:49.717 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:20:49.717 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:20:49.717 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:20:49.717 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:20:49.717 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:20:49.717 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:20:49.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:20:49.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:20:49.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:21:09.720 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:21:09.752 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:21:09.752 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:21:09.752 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:21:09.752 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:21:09.752 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:21:09.752 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:21:09.752 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:21:09.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:21:09.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.107 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:21:09.755 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:21:09.755 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:21:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:21:09.755 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:21:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:21:09.755 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:21:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:21:09.755 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:21:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:21:09.755 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:21:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:21:09.755 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:21:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:21:09.755 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:21:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:21:09.755 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:21:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:21:09.755 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:21:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:21:09.755 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:21:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:21:09.755 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:21:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:21:09.755 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:21:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:21:09.755 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:21:09.755 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:21:09.755 139328 ERROR oslo_service.periodic_task >2018-06-24 16:21:39.637 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:21:39.638 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:21:39.644 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:21:39.645 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:21:39.679 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:21:39.679 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:21:39.679 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:21:39.679 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:21:39.679 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:21:39.679 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:21:39.679 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:21:39.681 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:21:39.681 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:21:39.681 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:21:49.682 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:21:49.716 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:21:49.716 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:21:49.716 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:21:49.716 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:21:49.716 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:21:49.716 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:21:49.716 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:21:49.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:21:49.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:21:49.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:22:09.719 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:22:09.756 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:22:09.756 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:22:09.756 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:22:09.756 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:22:09.756 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:22:09.756 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:22:09.756 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:22:09.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:22:09.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:22:09.759 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:22:09.759 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:22:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:22:09.759 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:22:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:22:09.759 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:22:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:22:09.759 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:22:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:22:09.759 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:22:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:22:09.759 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:22:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:22:09.759 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:22:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:22:09.759 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:22:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:22:09.759 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:22:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:22:09.759 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:22:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:22:09.759 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:22:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:22:09.759 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:22:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:22:09.759 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:22:09.759 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:22:09.759 139328 ERROR oslo_service.periodic_task >2018-06-24 16:22:39.640 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:22:39.642 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:22:39.649 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:22:39.649 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:22:39.683 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:22:39.683 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:22:39.683 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:22:39.683 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:22:39.683 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:22:39.683 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:22:39.683 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:22:39.685 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:22:39.686 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:22:39.686 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:22:49.686 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:22:49.722 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:22:49.722 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:22:49.722 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:22:49.722 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:22:49.722 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:22:49.722 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:22:49.722 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:22:49.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:22:49.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:22:49.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:23:09.726 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:23:09.761 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:23:09.761 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:23:09.761 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:23:09.761 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:23:09.761 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:23:09.761 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:23:09.761 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:23:09.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:23:09.764 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:23:09.764 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:23:09.764 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:23:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:23:09.764 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:23:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:23:09.764 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:23:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:23:09.764 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:23:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:23:09.764 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:23:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:23:09.764 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:23:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:23:09.764 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:23:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:23:09.764 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:23:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:23:09.764 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:23:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:23:09.764 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:23:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:23:09.764 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:23:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:23:09.764 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:23:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:23:09.764 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:23:09.764 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:23:09.764 139328 ERROR oslo_service.periodic_task >2018-06-24 16:23:39.645 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:23:39.646 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:23:39.654 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:23:39.655 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:23:39.689 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:23:39.689 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:23:39.689 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:23:39.689 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:23:39.689 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:23:39.689 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:23:39.689 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:23:39.691 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:23:39.691 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:23:39.691 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:23:49.692 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:23:49.725 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:23:49.725 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:23:49.725 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:23:49.725 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:23:49.725 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:23:49.725 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:23:49.725 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:23:49.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:23:49.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:23:49.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:24:09.728 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:24:09.762 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:24:09.762 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:24:09.762 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:24:09.762 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:24:09.762 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:24:09.762 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:24:09.762 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:24:09.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:24:09.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:24:09.766 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:24:09.766 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:24:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:24:09.766 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:24:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:24:09.766 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:24:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:24:09.766 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:24:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:24:09.766 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:24:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:24:09.766 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:24:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:24:09.766 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:24:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:24:09.766 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:24:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:24:09.766 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:24:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:24:09.766 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:24:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:24:09.766 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:24:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:24:09.766 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:24:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:24:09.766 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:24:09.766 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:24:09.766 139328 ERROR oslo_service.periodic_task >2018-06-24 16:24:39.648 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:24:39.650 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:24:39.657 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:24:39.657 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:24:39.705 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:24:39.705 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:24:39.705 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:24:39.705 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:24:39.705 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:24:39.705 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:24:39.705 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:24:39.707 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:24:39.708 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.05 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:24:39.708 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:24:49.709 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:24:49.742 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:24:49.742 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:24:49.742 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:24:49.742 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:24:49.742 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:24:49.742 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:24:49.742 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:24:49.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:24:49.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.086 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:24:49.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:25:09.745 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:25:09.778 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:25:09.778 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:25:09.778 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:25:09.778 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:25:09.778 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:25:09.778 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:25:09.778 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:25:09.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:25:09.781 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.123 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:25:09.781 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:25:09.781 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:25:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:25:09.781 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:25:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:25:09.781 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:25:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:25:09.781 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:25:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:25:09.781 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:25:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:25:09.781 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:25:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:25:09.781 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:25:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:25:09.781 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:25:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:25:09.781 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:25:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:25:09.781 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:25:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:25:09.781 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:25:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:25:09.781 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:25:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:25:09.781 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:25:09.781 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:25:09.781 139328 ERROR oslo_service.periodic_task >2018-06-24 16:25:39.652 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:25:39.653 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:25:39.662 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:25:39.662 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:25:39.698 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:25:39.698 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:25:39.698 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:25:39.698 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:25:39.698 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:25:39.698 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:25:39.698 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:25:39.700 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:25:39.701 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:25:39.701 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:25:49.702 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:25:49.741 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:25:49.741 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:25:49.741 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:25:49.741 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:25:49.741 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:25:49.741 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:25:49.741 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:25:49.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:25:49.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.081 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:25:49.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:26:09.745 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:26:09.779 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:26:09.779 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:26:09.779 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:26:09.779 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:26:09.779 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:26:09.779 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:26:09.779 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:26:09.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:26:09.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.119 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:26:09.783 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:26:09.783 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:26:09.783 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:26:09.783 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:26:09.783 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:26:09.783 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:26:09.783 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:26:09.783 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:26:09.783 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:26:09.783 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:26:09.783 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:26:09.783 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:26:09.783 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:26:09.783 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:26:09.783 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:26:09.783 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:26:09.783 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:26:09.783 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:26:09.783 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:26:09.783 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:26:09.783 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:26:09.783 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:26:09.783 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:26:09.783 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:26:09.783 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:26:09.783 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:26:09.783 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:26:09.783 139328 ERROR oslo_service.periodic_task >2018-06-24 16:26:39.654 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:26:39.655 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:26:39.661 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:26:39.662 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:26:39.696 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:26:39.696 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:26:39.696 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:26:39.696 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:26:39.696 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:26:39.696 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:26:39.696 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:26:39.698 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:26:39.699 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:26:39.699 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:26:49.700 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:26:49.734 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:26:49.734 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:26:49.734 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:26:49.734 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:26:49.734 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:26:49.734 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:26:49.734 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:26:49.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:26:49.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:26:49.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:27:09.737 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:27:09.776 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:27:09.776 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:27:09.776 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:27:09.776 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:27:09.776 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:27:09.776 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:27:09.776 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:27:09.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:27:09.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:27:09.779 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:27:09.779 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:27:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:27:09.779 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:27:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:27:09.779 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:27:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:27:09.779 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:27:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:27:09.779 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:27:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:27:09.779 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:27:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:27:09.779 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:27:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:27:09.779 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:27:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:27:09.779 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:27:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:27:09.779 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:27:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:27:09.779 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:27:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:27:09.779 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:27:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:27:09.779 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:27:09.779 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:27:09.779 139328 ERROR oslo_service.periodic_task >2018-06-24 16:27:39.650 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:27:39.652 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:27:39.658 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:27:39.659 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:27:39.699 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:27:39.699 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:27:39.699 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:27:39.699 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:27:39.699 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:27:39.699 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:27:39.699 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:27:39.702 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:27:39.703 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.044 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:27:39.703 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:27:49.703 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:27:49.741 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:27:49.741 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:27:49.741 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:27:49.741 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:27:49.741 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:27:49.741 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:27:49.741 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:27:49.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:27:49.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.084 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:27:49.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:28:09.744 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:28:09.777 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:28:09.777 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:28:09.777 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:28:09.777 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:28:09.777 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:28:09.777 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:28:09.777 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:28:09.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:28:09.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.121 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:28:09.780 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:28:09.780 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:28:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:28:09.780 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:28:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:28:09.780 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:28:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:28:09.780 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:28:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:28:09.780 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:28:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:28:09.780 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:28:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:28:09.780 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:28:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:28:09.780 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:28:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:28:09.780 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:28:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:28:09.780 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:28:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:28:09.780 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:28:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:28:09.780 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:28:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:28:09.780 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:28:09.780 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:28:09.780 139328 ERROR oslo_service.periodic_task >2018-06-24 16:28:39.652 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:28:39.652 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:28:39.660 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:28:39.660 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:28:39.693 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:28:39.693 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:28:39.693 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:28:39.693 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:28:39.693 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:28:39.693 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:28:39.693 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:28:39.695 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:28:39.695 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:28:39.695 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:28:49.696 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:28:49.734 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:28:49.734 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:28:49.734 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:28:49.734 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:28:49.734 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:28:49.734 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:28:49.734 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:28:49.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:28:49.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:28:49.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:29:09.738 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:29:09.772 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:29:09.772 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:29:09.772 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:29:09.772 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:29:09.772 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:29:09.772 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:29:09.772 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:29:09.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:29:09.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:29:09.775 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:29:09.775 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:29:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:29:09.775 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:29:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:29:09.775 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:29:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:29:09.775 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:29:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:29:09.775 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:29:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:29:09.775 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:29:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:29:09.775 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:29:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:29:09.775 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:29:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:29:09.775 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:29:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:29:09.775 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:29:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:29:09.775 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:29:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:29:09.775 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:29:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:29:09.775 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:29:09.775 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:29:09.775 139328 ERROR oslo_service.periodic_task >2018-06-24 16:29:39.656 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:29:39.657 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:29:39.665 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:29:39.665 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:29:39.700 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:29:39.700 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:29:39.700 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:29:39.700 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:29:39.700 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:29:39.700 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:29:39.700 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:29:39.703 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:29:39.703 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:29:39.703 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:29:49.704 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:29:49.739 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:29:49.739 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:29:49.739 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:29:49.739 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:29:49.739 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:29:49.739 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:29:49.739 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:29:49.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:29:49.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:29:49.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:30:09.743 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:30:09.776 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:30:09.776 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:30:09.776 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:30:09.776 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:30:09.776 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:30:09.776 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:30:09.776 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:30:09.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:30:09.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:30:09.780 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:30:09.780 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:30:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:30:09.780 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:30:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:30:09.780 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:30:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:30:09.780 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:30:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:30:09.780 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:30:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:30:09.780 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:30:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:30:09.780 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:30:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:30:09.780 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:30:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:30:09.780 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:30:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:30:09.780 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:30:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:30:09.780 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:30:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:30:09.780 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:30:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:30:09.780 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:30:09.780 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:30:09.780 139328 ERROR oslo_service.periodic_task >2018-06-24 16:30:39.652 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:30:39.653 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:30:39.660 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:30:39.661 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:30:39.700 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:30:39.700 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:30:39.700 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:30:39.700 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:30:39.700 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:30:39.700 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:30:39.700 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:30:39.703 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:30:39.703 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.042 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:30:39.704 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:30:49.704 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:30:49.745 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:30:49.745 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:30:49.745 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:30:49.745 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:30:49.745 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:30:49.745 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:30:49.745 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:30:49.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:30:49.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.087 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:30:49.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:31:09.749 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:31:09.781 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:31:09.781 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:31:09.781 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:31:09.781 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:31:09.781 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:31:09.781 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:31:09.781 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:31:09.783 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:31:09.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.123 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:31:09.784 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:31:09.784 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:31:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:31:09.784 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:31:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:31:09.784 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:31:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:31:09.784 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:31:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:31:09.784 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:31:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:31:09.784 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:31:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:31:09.784 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:31:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:31:09.784 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:31:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:31:09.784 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:31:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:31:09.784 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:31:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:31:09.784 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:31:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:31:09.784 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:31:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:31:09.784 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:31:09.784 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:31:09.784 139328 ERROR oslo_service.periodic_task >2018-06-24 16:31:39.655 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:31:39.656 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:31:39.670 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:31:39.671 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:31:39.704 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:31:39.704 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:31:39.704 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:31:39.704 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:31:39.704 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:31:39.704 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:31:39.704 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:31:39.705 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:31:39.706 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:31:39.706 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:31:49.707 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:31:49.739 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:31:49.739 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:31:49.739 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:31:49.739 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:31:49.739 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:31:49.739 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:31:49.739 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:31:49.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:31:49.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.07 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:31:49.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:32:09.742 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:32:09.776 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:32:09.776 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:32:09.776 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:32:09.776 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:32:09.776 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:32:09.776 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:32:09.776 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:32:09.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:32:09.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.107 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:32:09.778 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:32:09.778 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:32:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:32:09.778 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:32:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:32:09.778 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:32:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:32:09.778 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:32:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:32:09.778 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:32:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:32:09.778 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:32:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:32:09.778 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:32:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:32:09.778 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:32:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:32:09.778 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:32:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:32:09.778 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:32:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:32:09.778 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:32:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:32:09.778 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:32:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:32:09.778 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:32:09.778 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:32:09.778 139328 ERROR oslo_service.periodic_task >2018-06-24 16:32:39.660 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:32:39.661 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:32:39.669 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:32:39.669 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:32:39.708 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:32:39.708 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:32:39.708 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:32:39.708 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:32:39.708 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:32:39.708 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:32:39.708 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:32:39.710 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:32:39.710 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:32:39.711 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:32:49.711 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:32:49.742 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:32:49.742 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:32:49.742 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:32:49.742 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:32:49.742 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:32:49.742 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:32:49.742 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:32:49.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:32:49.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:32:49.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:33:09.745 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:33:09.781 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:33:09.781 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:33:09.781 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:33:09.781 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:33:09.781 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:33:09.781 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:33:09.781 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:33:09.783 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:33:09.783 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:33:09.784 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:33:09.784 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:33:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:33:09.784 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:33:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:33:09.784 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:33:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:33:09.784 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:33:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:33:09.784 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:33:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:33:09.784 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:33:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:33:09.784 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:33:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:33:09.784 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:33:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:33:09.784 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:33:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:33:09.784 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:33:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:33:09.784 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:33:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:33:09.784 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:33:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:33:09.784 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:33:09.784 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:33:09.784 139328 ERROR oslo_service.periodic_task >2018-06-24 16:33:39.655 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:33:39.656 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:33:39.663 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:33:39.663 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:33:39.697 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:33:39.697 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:33:39.697 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:33:39.697 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:33:39.697 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:33:39.697 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:33:39.697 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:33:39.699 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:33:39.699 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:33:39.700 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:33:49.700 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:33:49.736 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:33:49.736 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:33:49.736 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:33:49.736 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:33:49.736 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:33:49.736 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:33:49.736 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:33:49.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:33:49.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:33:49.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:34:09.739 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:34:09.776 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:34:09.776 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:34:09.776 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:34:09.776 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:34:09.776 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:34:09.776 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:34:09.776 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:34:09.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:34:09.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:34:09.778 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:34:09.778 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:34:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:34:09.778 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:34:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:34:09.778 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:34:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:34:09.778 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:34:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:34:09.778 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:34:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:34:09.778 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:34:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:34:09.778 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:34:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:34:09.778 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:34:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:34:09.778 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:34:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:34:09.778 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:34:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:34:09.778 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:34:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:34:09.778 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:34:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:34:09.778 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:34:09.778 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:34:09.778 139328 ERROR oslo_service.periodic_task >2018-06-24 16:34:39.660 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:34:39.660 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:34:39.667 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:34:39.668 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:34:39.701 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:34:39.701 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:34:39.701 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:34:39.701 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:34:39.701 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:34:39.701 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:34:39.701 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:34:39.703 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:34:39.703 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:34:39.704 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:34:49.704 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:34:49.736 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:34:49.736 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:34:49.736 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:34:49.736 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:34:49.736 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:34:49.736 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:34:49.736 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:34:49.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:34:49.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:34:49.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:35:09.740 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:35:09.774 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:35:09.774 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:35:09.774 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:35:09.774 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:35:09.774 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:35:09.774 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:35:09.774 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:35:09.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:35:09.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:35:09.777 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:35:09.777 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:35:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:35:09.777 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:35:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:35:09.777 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:35:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:35:09.777 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:35:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:35:09.777 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:35:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:35:09.777 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:35:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:35:09.777 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:35:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:35:09.777 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:35:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:35:09.777 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:35:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:35:09.777 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:35:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:35:09.777 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:35:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:35:09.777 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:35:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:35:09.777 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:35:09.777 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:35:09.777 139328 ERROR oslo_service.periodic_task >2018-06-24 16:35:39.658 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:35:39.659 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:35:39.667 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:35:39.668 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:35:39.701 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:35:39.701 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:35:39.701 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:35:39.701 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:35:39.701 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:35:39.701 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:35:39.701 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:35:39.703 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:35:39.703 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:35:39.704 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:35:49.704 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:35:49.738 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:35:49.738 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:35:49.738 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:35:49.738 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:35:49.738 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:35:49.738 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:35:49.738 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:35:49.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:35:49.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:35:49.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:36:09.741 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:36:09.775 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:36:09.775 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:36:09.775 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:36:09.775 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:36:09.775 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:36:09.775 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:36:09.775 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:36:09.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:36:09.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:36:09.778 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:36:09.778 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:36:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:36:09.778 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:36:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:36:09.778 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:36:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:36:09.778 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:36:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:36:09.778 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:36:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:36:09.778 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:36:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:36:09.778 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:36:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:36:09.778 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:36:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:36:09.778 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:36:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:36:09.778 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:36:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:36:09.778 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:36:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:36:09.778 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:36:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:36:09.778 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:36:09.778 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:36:09.778 139328 ERROR oslo_service.periodic_task >2018-06-24 16:36:39.660 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:36:39.661 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:36:39.669 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:36:39.669 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:36:39.705 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:36:39.705 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:36:39.705 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:36:39.705 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:36:39.705 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:36:39.705 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:36:39.705 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:36:39.708 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:36:39.708 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:36:39.708 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:36:49.709 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:36:49.743 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:36:49.743 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:36:49.743 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:36:49.743 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:36:49.743 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:36:49.743 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:36:49.743 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:36:49.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:36:49.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:36:49.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:37:09.746 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:37:09.780 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:37:09.780 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:37:09.780 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:37:09.780 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:37:09.780 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:37:09.780 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:37:09.780 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:37:09.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:37:09.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:37:09.782 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:37:09.782 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:37:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:37:09.782 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:37:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:37:09.782 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:37:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:37:09.782 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:37:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:37:09.782 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:37:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:37:09.782 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:37:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:37:09.782 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:37:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:37:09.782 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:37:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:37:09.782 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:37:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:37:09.782 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:37:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:37:09.782 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:37:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:37:09.782 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:37:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:37:09.782 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:37:09.782 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:37:09.782 139328 ERROR oslo_service.periodic_task >2018-06-24 16:37:39.664 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:37:39.665 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:37:39.671 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:37:39.672 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:37:39.703 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:37:39.703 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:37:39.703 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:37:39.703 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:37:39.703 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:37:39.703 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:37:39.703 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:37:39.705 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:37:39.706 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:37:39.706 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:37:49.706 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:37:49.743 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:37:49.743 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:37:49.743 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:37:49.743 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:37:49.743 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:37:49.743 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:37:49.743 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:37:49.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:37:49.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:37:49.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:38:09.747 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:38:09.782 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:38:09.782 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:38:09.782 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:38:09.782 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:38:09.782 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:38:09.782 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:38:09.782 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:38:09.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:38:09.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:38:09.785 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:38:09.785 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:38:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:38:09.785 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:38:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:38:09.785 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:38:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:38:09.785 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:38:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:38:09.785 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:38:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:38:09.785 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:38:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:38:09.785 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:38:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:38:09.785 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:38:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:38:09.785 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:38:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:38:09.785 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:38:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:38:09.785 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:38:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:38:09.785 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:38:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:38:09.785 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:38:09.785 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:38:09.785 139328 ERROR oslo_service.periodic_task >2018-06-24 16:38:39.666 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:38:39.667 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:38:39.676 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:38:39.677 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:38:39.715 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:38:39.715 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:38:39.715 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:38:39.715 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:38:39.715 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:38:39.715 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:38:39.715 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:38:39.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:38:39.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:38:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:38:49.718 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:38:49.753 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:38:49.753 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:38:49.753 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:38:49.753 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:38:49.753 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:38:49.753 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:38:49.753 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:38:49.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:38:49.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:38:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:39:09.757 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:39:09.790 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:39:09.790 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:39:09.790 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:39:09.790 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:39:09.790 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:39:09.790 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:39:09.790 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:39:09.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:39:09.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:39:09.793 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:39:09.793 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:39:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:39:09.793 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:39:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:39:09.793 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:39:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:39:09.793 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:39:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:39:09.793 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:39:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:39:09.793 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:39:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:39:09.793 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:39:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:39:09.793 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:39:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:39:09.793 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:39:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:39:09.793 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:39:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:39:09.793 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:39:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:39:09.793 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:39:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:39:09.793 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:39:09.793 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:39:09.793 139328 ERROR oslo_service.periodic_task >2018-06-24 16:39:39.665 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:39:39.666 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:39:39.673 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:39:39.674 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:39:39.707 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:39:39.707 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:39:39.707 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:39:39.707 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:39:39.707 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:39:39.707 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:39:39.707 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:39:39.709 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:39:39.709 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:39:39.710 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:39:49.712 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:39:49.760 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:39:49.760 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:39:49.760 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:39:49.760 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:39:49.760 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:39:49.760 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:39:49.760 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:39:49.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:39:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.094 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:39:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:40:09.769 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:40:09.799 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:40:09.799 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:40:09.799 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:40:09.799 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:40:09.799 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:40:09.799 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:40:09.799 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:40:09.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:40:09.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.128 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:40:09.802 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:40:09.802 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:40:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:40:09.802 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:40:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:40:09.802 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:40:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:40:09.802 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:40:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:40:09.802 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:40:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:40:09.802 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:40:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:40:09.802 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:40:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:40:09.802 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:40:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:40:09.802 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:40:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:40:09.802 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:40:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:40:09.802 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:40:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:40:09.802 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:40:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:40:09.802 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:40:09.802 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:40:09.802 139328 ERROR oslo_service.periodic_task >2018-06-24 16:40:39.664 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:40:39.665 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:40:39.678 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:40:39.678 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:40:39.711 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:40:39.711 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:40:39.711 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:40:39.711 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:40:39.711 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:40:39.711 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:40:39.711 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:40:39.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:40:39.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:40:39.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:40:49.715 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:40:49.753 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:40:49.753 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:40:49.753 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:40:49.753 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:40:49.753 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:40:49.753 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:40:49.753 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:40:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:40:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:40:49.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:41:09.759 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:41:09.791 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:41:09.791 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:41:09.791 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:41:09.791 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:41:09.791 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:41:09.791 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:41:09.791 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:41:09.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:41:09.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:41:09.793 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:41:09.793 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:41:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:41:09.793 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:41:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:41:09.793 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:41:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:41:09.793 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:41:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:41:09.793 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:41:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:41:09.793 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:41:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:41:09.793 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:41:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:41:09.793 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:41:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:41:09.793 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:41:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:41:09.793 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:41:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:41:09.793 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:41:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:41:09.793 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:41:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:41:09.793 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:41:09.793 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:41:09.793 139328 ERROR oslo_service.periodic_task >2018-06-24 16:41:39.665 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:41:39.665 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:41:39.672 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:41:39.673 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:41:39.709 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:41:39.709 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:41:39.709 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:41:39.709 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:41:39.709 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:41:39.709 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:41:39.709 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:41:39.711 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:41:39.711 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:41:39.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:41:49.712 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:41:49.746 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:41:49.746 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:41:49.746 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:41:49.746 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:41:49.746 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:41:49.746 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:41:49.746 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:41:49.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:41:49.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:41:49.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:42:09.749 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:42:09.783 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:42:09.783 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:42:09.783 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:42:09.783 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:42:09.783 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:42:09.783 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:42:09.783 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:42:09.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:42:09.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:42:09.786 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:42:09.786 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:42:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:42:09.786 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:42:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:42:09.786 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:42:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:42:09.786 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:42:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:42:09.786 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:42:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:42:09.786 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:42:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:42:09.786 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:42:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:42:09.786 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:42:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:42:09.786 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:42:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:42:09.786 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:42:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:42:09.786 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:42:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:42:09.786 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:42:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:42:09.786 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:42:09.786 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:42:09.786 139328 ERROR oslo_service.periodic_task >2018-06-24 16:42:39.668 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:42:39.669 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:42:39.676 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:42:39.676 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:42:39.709 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:42:39.709 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:42:39.709 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:42:39.709 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:42:39.709 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:42:39.709 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:42:39.709 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:42:39.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:42:39.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:42:39.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:42:49.713 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:42:49.746 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:42:49.746 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:42:49.746 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:42:49.746 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:42:49.746 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:42:49.746 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:42:49.746 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:42:49.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:42:49.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:42:49.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:43:09.749 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:43:09.783 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:43:09.783 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:43:09.783 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:43:09.783 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:43:09.783 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:43:09.783 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:43:09.783 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:43:09.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:43:09.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:43:09.786 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:43:09.786 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:43:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:43:09.786 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:43:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:43:09.786 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:43:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:43:09.786 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:43:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:43:09.786 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:43:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:43:09.786 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:43:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:43:09.786 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:43:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:43:09.786 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:43:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:43:09.786 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:43:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:43:09.786 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:43:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:43:09.786 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:43:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:43:09.786 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:43:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:43:09.786 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:43:09.786 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:43:09.786 139328 ERROR oslo_service.periodic_task >2018-06-24 16:43:39.667 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:43:39.668 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:43:39.676 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:43:39.676 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:43:39.724 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:43:39.724 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:43:39.724 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:43:39.724 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:43:39.724 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:43:39.724 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:43:39.724 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:43:39.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:43:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.051 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:43:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:43:49.729 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:43:49.767 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:43:49.767 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:43:49.767 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:43:49.767 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:43:49.767 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:43:49.767 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:43:49.767 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:43:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:43:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.092 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:43:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:44:09.770 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:44:09.803 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:44:09.803 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:44:09.803 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:44:09.803 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:44:09.803 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:44:09.803 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:44:09.803 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:44:09.805 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:44:09.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.129 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:44:09.806 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:44:09.806 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:44:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:44:09.806 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:44:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:44:09.806 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:44:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:44:09.806 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:44:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:44:09.806 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:44:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:44:09.806 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:44:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:44:09.806 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:44:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:44:09.806 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:44:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:44:09.806 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:44:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:44:09.806 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:44:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:44:09.806 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:44:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:44:09.806 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:44:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:44:09.806 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:44:09.806 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:44:09.806 139328 ERROR oslo_service.periodic_task >2018-06-24 16:44:39.667 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:44:39.669 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:44:39.678 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:44:39.678 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:44:39.712 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:44:39.712 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:44:39.712 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:44:39.712 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:44:39.712 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:44:39.712 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:44:39.712 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:44:39.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:44:39.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:44:39.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:44:49.716 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:44:49.749 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:44:49.749 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:44:49.749 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:44:49.749 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:44:49.749 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:44:49.749 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:44:49.749 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:44:49.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:44:49.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:44:49.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:45:09.752 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:45:09.788 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:45:09.788 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:45:09.788 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:45:09.788 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:45:09.788 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:45:09.788 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:45:09.788 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:45:09.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:45:09.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:45:09.791 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:45:09.791 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:45:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:45:09.791 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:45:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:45:09.791 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:45:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:45:09.791 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:45:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:45:09.791 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:45:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:45:09.791 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:45:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:45:09.791 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:45:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:45:09.791 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:45:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:45:09.791 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:45:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:45:09.791 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:45:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:45:09.791 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:45:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:45:09.791 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:45:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:45:09.791 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:45:09.791 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:45:09.791 139328 ERROR oslo_service.periodic_task >2018-06-24 16:45:39.663 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:45:39.663 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:45:39.671 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:45:39.672 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:45:39.715 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:45:39.715 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:45:39.715 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:45:39.715 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:45:39.715 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:45:39.715 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:45:39.715 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:45:39.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:45:39.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.046 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:45:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:45:49.718 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:45:49.752 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:45:49.752 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:45:49.752 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:45:49.752 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:45:49.752 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:45:49.752 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:45:49.752 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:45:49.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:45:49.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.083 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:45:49.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:46:09.756 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:46:09.788 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:46:09.788 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:46:09.788 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:46:09.788 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:46:09.788 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:46:09.788 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:46:09.788 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:46:09.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:46:09.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.118 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:46:09.790 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:46:09.790 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:46:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:46:09.790 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:46:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:46:09.790 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:46:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:46:09.790 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:46:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:46:09.790 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:46:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:46:09.790 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:46:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:46:09.790 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:46:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:46:09.790 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:46:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:46:09.790 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:46:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:46:09.790 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:46:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:46:09.790 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:46:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:46:09.790 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:46:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:46:09.790 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:46:09.790 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:46:09.790 139328 ERROR oslo_service.periodic_task >2018-06-24 16:46:39.662 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:46:39.663 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:46:39.670 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:46:39.670 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:46:39.703 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:46:39.703 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:46:39.703 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:46:39.703 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:46:39.703 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:46:39.703 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:46:39.703 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:46:39.705 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:46:39.706 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:46:39.706 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:46:49.707 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:46:49.740 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:46:49.740 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:46:49.740 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:46:49.740 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:46:49.740 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:46:49.740 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:46:49.740 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:46:49.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:46:49.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:46:49.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:47:09.743 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:47:09.774 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:47:09.774 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:47:09.774 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:47:09.774 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:47:09.774 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:47:09.774 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:47:09.774 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:47:09.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:47:09.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.106 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:47:09.777 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:47:09.777 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:47:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:47:09.777 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:47:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:47:09.777 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:47:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:47:09.777 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:47:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:47:09.777 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:47:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:47:09.777 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:47:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:47:09.777 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:47:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:47:09.777 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:47:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:47:09.777 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:47:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:47:09.777 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:47:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:47:09.777 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:47:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:47:09.777 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:47:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:47:09.777 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:47:09.777 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:47:09.777 139328 ERROR oslo_service.periodic_task >2018-06-24 16:47:39.659 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:47:39.660 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:47:39.668 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:47:39.668 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:47:39.702 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:47:39.702 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:47:39.702 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:47:39.702 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:47:39.702 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:47:39.702 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:47:39.702 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:47:39.704 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:47:39.704 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:47:39.705 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:47:49.705 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:47:49.743 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:47:49.743 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:47:49.743 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:47:49.743 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:47:49.743 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:47:49.743 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:47:49.743 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:47:49.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:47:49.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:47:49.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:48:09.746 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:48:09.780 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:48:09.780 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:48:09.780 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:48:09.780 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:48:09.780 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:48:09.780 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:48:09.780 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:48:09.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:48:09.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:48:09.782 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:48:09.782 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:48:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:48:09.782 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:48:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:48:09.782 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:48:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:48:09.782 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:48:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:48:09.782 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:48:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:48:09.782 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:48:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:48:09.782 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:48:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:48:09.782 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:48:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:48:09.782 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:48:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:48:09.782 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:48:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:48:09.782 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:48:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:48:09.782 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:48:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:48:09.782 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:48:09.782 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:48:09.782 139328 ERROR oslo_service.periodic_task >2018-06-24 16:48:39.654 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:48:39.655 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:48:39.662 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:48:39.663 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:48:39.703 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:48:39.703 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:48:39.703 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:48:39.703 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:48:39.703 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:48:39.703 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:48:39.703 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:48:39.706 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:48:39.706 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.043 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:48:39.707 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:48:49.707 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:48:49.741 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:48:49.741 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:48:49.741 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:48:49.741 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:48:49.741 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:48:49.741 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:48:49.741 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:48:49.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:48:49.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.08 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:48:49.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:49:09.744 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:49:09.777 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:49:09.777 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:49:09.777 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:49:09.777 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:49:09.777 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:49:09.777 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:49:09.777 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:49:09.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:49:09.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:49:09.780 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:49:09.780 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:49:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:49:09.780 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:49:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:49:09.780 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:49:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:49:09.780 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:49:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:49:09.780 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:49:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:49:09.780 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:49:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:49:09.780 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:49:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:49:09.780 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:49:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:49:09.780 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:49:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:49:09.780 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:49:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:49:09.780 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:49:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:49:09.780 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:49:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:49:09.780 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:49:09.780 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:49:09.780 139328 ERROR oslo_service.periodic_task >2018-06-24 16:49:39.651 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:49:39.652 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:49:39.662 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:49:39.662 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:49:39.696 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:49:39.696 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:49:39.696 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:49:39.696 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:49:39.696 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:49:39.696 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:49:39.696 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:49:39.698 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:49:39.699 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:49:39.699 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:49:49.700 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:49:49.736 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:49:49.736 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:49:49.736 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:49:49.736 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:49:49.736 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:49:49.736 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:49:49.736 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:49:49.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:49:49.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:49:49.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:50:09.740 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:50:09.773 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:50:09.773 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:50:09.773 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:50:09.773 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:50:09.773 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:50:09.773 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:50:09.773 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:50:09.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:50:09.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:50:09.776 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:50:09.776 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:50:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:50:09.776 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:50:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:50:09.776 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:50:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:50:09.776 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:50:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:50:09.776 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:50:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:50:09.776 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:50:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:50:09.776 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:50:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:50:09.776 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:50:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:50:09.776 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:50:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:50:09.776 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:50:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:50:09.776 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:50:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:50:09.776 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:50:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:50:09.776 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:50:09.776 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:50:09.776 139328 ERROR oslo_service.periodic_task >2018-06-24 16:50:39.647 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:50:39.648 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:50:39.656 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:50:39.657 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:50:39.693 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:50:39.693 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:50:39.693 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:50:39.693 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:50:39.693 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:50:39.693 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:50:39.693 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:50:39.695 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:50:39.695 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:50:39.696 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:50:49.696 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:50:49.731 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:50:49.731 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:50:49.731 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:50:49.731 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:50:49.731 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:50:49.731 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:50:49.731 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:50:49.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:50:49.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:50:49.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:51:09.734 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:51:09.766 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:51:09.766 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:51:09.766 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:51:09.766 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:51:09.766 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:51:09.766 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:51:09.766 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:51:09.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:51:09.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:51:09.769 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:51:09.769 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:51:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:51:09.769 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:51:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:51:09.769 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:51:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:51:09.769 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:51:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:51:09.769 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:51:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:51:09.769 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:51:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:51:09.769 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:51:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:51:09.769 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:51:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:51:09.769 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:51:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:51:09.769 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:51:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:51:09.769 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:51:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:51:09.769 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:51:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:51:09.769 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:51:09.769 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:51:09.769 139328 ERROR oslo_service.periodic_task >2018-06-24 16:51:39.650 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:51:39.651 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:51:39.662 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:51:39.662 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:51:39.696 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:51:39.696 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:51:39.696 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:51:39.696 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:51:39.696 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:51:39.696 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:51:39.696 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:51:39.698 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:51:39.699 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:51:39.699 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:51:49.700 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:51:49.731 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:51:49.731 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:51:49.731 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:51:49.731 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:51:49.731 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:51:49.731 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:51:49.731 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:51:49.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:51:49.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:51:49.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:52:09.734 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:52:09.768 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:52:09.768 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:52:09.768 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:52:09.768 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:52:09.768 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:52:09.768 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:52:09.768 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:52:09.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:52:09.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:52:09.771 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:52:09.771 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:52:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:52:09.771 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:52:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:52:09.771 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:52:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:52:09.771 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:52:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:52:09.771 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:52:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:52:09.771 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:52:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:52:09.771 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:52:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:52:09.771 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:52:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:52:09.771 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:52:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:52:09.771 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:52:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:52:09.771 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:52:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:52:09.771 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:52:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:52:09.771 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:52:09.771 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:52:09.771 139328 ERROR oslo_service.periodic_task >2018-06-24 16:52:39.653 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:52:39.654 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:52:39.663 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:52:39.664 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:52:39.697 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:52:39.697 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:52:39.697 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:52:39.697 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:52:39.697 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:52:39.697 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:52:39.697 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:52:39.698 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:52:39.699 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:52:39.699 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:52:49.700 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:52:49.732 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:52:49.732 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:52:49.732 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:52:49.732 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:52:49.732 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:52:49.732 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:52:49.732 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:52:49.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:52:49.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:52:49.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:53:09.735 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:53:09.790 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:53:09.790 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:53:09.790 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:53:09.790 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:53:09.790 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:53:09.790 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:53:09.790 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:53:09.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:53:09.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.129 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:53:09.793 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:53:09.793 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:53:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:53:09.793 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:53:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:53:09.793 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:53:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:53:09.793 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:53:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:53:09.793 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:53:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:53:09.793 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:53:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:53:09.793 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:53:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:53:09.793 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:53:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:53:09.793 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:53:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:53:09.793 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:53:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:53:09.793 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:53:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:53:09.793 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:53:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:53:09.793 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:53:09.793 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:53:09.793 139328 ERROR oslo_service.periodic_task >2018-06-24 16:53:39.654 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:53:39.655 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:53:39.669 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:53:39.670 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:53:39.705 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:53:39.705 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:53:39.705 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:53:39.705 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:53:39.705 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:53:39.705 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:53:39.705 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:53:39.707 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:53:39.708 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:53:39.708 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:53:49.709 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:53:49.741 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:53:49.741 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:53:49.741 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:53:49.741 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:53:49.741 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:53:49.741 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:53:49.741 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:53:49.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:53:49.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:53:49.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:54:09.744 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:54:09.781 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:54:09.781 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:54:09.781 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:54:09.781 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:54:09.781 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:54:09.781 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:54:09.781 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:54:09.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:54:09.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:54:09.786 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:54:09.786 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:54:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:54:09.786 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:54:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:54:09.786 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:54:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:54:09.786 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:54:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:54:09.786 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:54:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:54:09.786 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:54:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:54:09.786 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:54:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:54:09.786 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:54:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:54:09.786 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:54:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:54:09.786 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:54:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:54:09.786 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:54:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:54:09.786 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:54:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:54:09.786 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:54:09.786 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:54:09.786 139328 ERROR oslo_service.periodic_task >2018-06-24 16:54:39.657 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:54:39.658 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:54:39.691 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:54:39.691 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:54:39.691 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:54:39.691 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:54:39.691 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:54:39.691 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:54:39.691 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:54:39.693 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:54:39.693 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:54:39.693 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:54:49.694 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:54:49.727 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:54:49.727 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:54:49.727 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:54:49.727 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:54:49.727 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:54:49.727 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:54:49.727 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:54:49.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:54:49.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.07 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:54:49.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:55:09.730 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:55:09.763 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:55:09.763 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:55:09.763 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:55:09.763 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:55:09.763 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:55:09.763 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:55:09.763 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:55:09.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:55:09.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.107 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:55:09.766 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:55:09.766 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:55:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:55:09.766 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:55:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:55:09.766 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:55:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:55:09.766 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:55:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:55:09.766 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:55:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:55:09.766 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:55:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:55:09.766 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:55:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:55:09.766 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:55:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:55:09.766 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:55:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:55:09.766 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:55:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:55:09.766 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:55:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:55:09.766 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:55:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:55:09.766 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:55:09.766 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:55:09.766 139328 ERROR oslo_service.periodic_task >2018-06-24 16:55:39.657 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:55:39.659 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:55:39.665 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:55:39.666 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:55:39.706 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:55:39.706 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:55:39.706 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:55:39.706 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:55:39.706 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:55:39.706 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:55:39.706 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:55:39.708 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:55:39.709 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.043 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:55:39.709 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:55:49.710 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:55:49.743 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:55:49.743 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:55:49.743 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:55:49.743 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:55:49.743 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:55:49.743 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:55:49.743 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:55:49.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:55:49.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.08 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:55:49.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:56:09.746 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:56:09.779 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:56:09.779 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:56:09.779 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:56:09.779 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:56:09.779 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:56:09.779 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:56:09.779 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:56:09.781 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:56:09.781 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:56:09.782 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:56:09.782 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:56:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:56:09.782 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:56:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:56:09.782 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:56:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:56:09.782 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:56:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:56:09.782 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:56:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:56:09.782 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:56:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:56:09.782 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:56:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:56:09.782 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:56:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:56:09.782 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:56:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:56:09.782 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:56:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:56:09.782 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:56:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:56:09.782 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:56:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:56:09.782 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:56:09.782 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:56:09.782 139328 ERROR oslo_service.periodic_task >2018-06-24 16:56:39.653 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:56:39.654 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:56:39.661 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:56:39.661 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:56:39.695 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:56:39.695 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:56:39.695 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:56:39.695 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:56:39.695 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:56:39.695 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:56:39.695 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:56:39.697 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:56:39.697 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:56:39.697 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:56:49.698 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:56:49.731 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:56:49.731 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:56:49.731 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:56:49.731 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:56:49.731 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:56:49.731 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:56:49.731 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:56:49.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:56:49.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:56:49.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:57:09.735 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:57:09.772 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:57:09.772 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:57:09.772 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:57:09.772 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:57:09.772 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:57:09.772 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:57:09.772 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:57:09.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:57:09.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:57:09.775 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:57:09.775 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:57:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:57:09.775 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:57:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:57:09.775 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:57:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:57:09.775 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:57:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:57:09.775 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:57:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:57:09.775 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:57:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:57:09.775 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:57:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:57:09.775 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:57:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:57:09.775 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:57:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:57:09.775 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:57:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:57:09.775 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:57:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:57:09.775 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:57:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:57:09.775 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:57:09.775 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:57:09.775 139328 ERROR oslo_service.periodic_task >2018-06-24 16:57:39.656 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:57:39.657 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:57:39.665 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:57:39.665 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:57:39.698 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:57:39.698 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:57:39.698 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:57:39.698 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:57:39.698 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:57:39.698 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:57:39.698 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:57:39.700 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:57:39.700 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:57:39.701 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:57:49.701 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:57:49.735 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:57:49.735 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:57:49.735 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:57:49.735 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:57:49.735 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:57:49.735 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:57:49.735 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:57:49.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:57:49.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:57:49.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:58:09.739 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:58:09.771 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:58:09.771 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:58:09.771 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:58:09.771 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:58:09.771 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:58:09.771 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:58:09.771 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:58:09.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:58:09.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:58:09.774 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:58:09.774 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:58:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:58:09.774 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:58:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:58:09.774 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:58:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:58:09.774 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:58:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:58:09.774 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:58:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:58:09.774 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:58:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:58:09.774 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:58:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:58:09.774 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:58:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:58:09.774 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:58:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:58:09.774 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:58:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:58:09.774 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:58:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:58:09.774 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:58:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:58:09.774 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:58:09.774 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:58:09.774 139328 ERROR oslo_service.periodic_task >2018-06-24 16:58:39.655 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:58:39.656 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:58:39.663 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:58:39.663 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:58:39.696 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:58:39.696 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:58:39.696 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:58:39.696 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:58:39.696 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:58:39.696 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:58:39.696 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:58:39.699 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:58:39.699 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:58:39.699 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:58:49.700 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:58:49.734 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:58:49.734 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:58:49.734 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:58:49.734 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:58:49.734 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:58:49.734 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:58:49.734 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:58:49.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:58:49.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:58:49.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:59:09.738 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:59:09.771 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:59:09.771 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:59:09.771 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:59:09.771 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:59:09.771 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:59:09.771 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:59:09.771 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:59:09.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:59:09.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:59:09.773 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:59:09.773 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 16:59:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 16:59:09.773 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 16:59:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 16:59:09.773 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 16:59:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 16:59:09.773 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 16:59:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 16:59:09.773 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 16:59:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 16:59:09.773 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 16:59:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 16:59:09.773 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 16:59:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 16:59:09.773 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 16:59:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 16:59:09.773 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 16:59:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 16:59:09.773 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 16:59:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 16:59:09.773 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 16:59:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 16:59:09.773 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 16:59:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 16:59:09.773 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 16:59:09.773 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 16:59:09.773 139328 ERROR oslo_service.periodic_task >2018-06-24 16:59:39.655 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:59:39.656 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 16:59:39.662 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 16:59:39.663 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:59:39.695 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:59:39.695 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:59:39.695 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:59:39.695 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:59:39.695 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:59:39.695 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:59:39.695 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:59:39.698 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:59:39.699 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:59:39.699 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 16:59:49.700 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 16:59:49.732 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 16:59:49.732 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 16:59:49.732 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 16:59:49.732 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 16:59:49.732 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 16:59:49.732 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 16:59:49.732 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 16:59:49.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 16:59:49.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 16:59:49.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:00:09.735 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:00:09.768 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:00:09.768 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:00:09.768 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:00:09.768 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:00:09.768 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:00:09.768 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:00:09.768 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:00:09.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:00:09.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:00:09.771 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:00:09.771 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:00:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:00:09.771 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:00:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:00:09.771 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:00:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:00:09.771 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:00:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:00:09.771 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:00:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:00:09.771 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:00:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:00:09.771 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:00:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:00:09.771 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:00:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:00:09.771 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:00:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:00:09.771 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:00:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:00:09.771 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:00:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:00:09.771 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:00:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:00:09.771 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:00:09.771 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:00:09.771 139328 ERROR oslo_service.periodic_task >2018-06-24 17:00:39.652 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:00:39.653 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:00:39.665 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:00:39.665 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:00:39.701 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:00:39.701 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:00:39.701 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:00:39.701 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:00:39.701 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:00:39.701 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:00:39.701 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:00:39.703 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:00:39.703 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:00:39.704 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:00:49.704 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:00:49.736 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:00:49.736 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:00:49.736 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:00:49.736 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:00:49.736 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:00:49.736 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:00:49.736 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:00:49.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:00:49.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:00:49.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:01:09.739 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:01:09.776 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:01:09.776 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:01:09.776 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:01:09.776 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:01:09.776 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:01:09.776 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:01:09.776 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:01:09.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:01:09.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:01:09.779 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:01:09.779 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:01:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:01:09.779 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:01:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:01:09.779 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:01:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:01:09.779 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:01:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:01:09.779 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:01:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:01:09.779 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:01:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:01:09.779 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:01:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:01:09.779 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:01:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:01:09.779 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:01:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:01:09.779 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:01:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:01:09.779 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:01:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:01:09.779 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:01:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:01:09.779 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:01:09.779 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:01:09.779 139328 ERROR oslo_service.periodic_task >2018-06-24 17:01:39.650 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:01:39.651 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:01:39.658 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:01:39.658 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:01:39.692 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:01:39.692 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:01:39.692 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:01:39.692 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:01:39.692 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:01:39.692 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:01:39.692 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:01:39.694 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:01:39.694 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:01:39.695 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:01:49.695 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:01:49.730 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:01:49.730 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:01:49.730 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:01:49.730 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:01:49.730 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:01:49.730 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:01:49.730 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:01:49.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:01:49.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:01:49.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:02:09.733 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:02:09.771 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:02:09.771 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:02:09.771 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:02:09.771 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:02:09.771 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:02:09.771 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:02:09.771 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:02:09.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:02:09.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:02:09.773 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:02:09.773 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:02:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:02:09.773 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:02:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:02:09.773 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:02:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:02:09.773 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:02:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:02:09.773 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:02:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:02:09.773 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:02:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:02:09.773 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:02:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:02:09.773 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:02:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:02:09.773 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:02:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:02:09.773 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:02:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:02:09.773 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:02:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:02:09.773 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:02:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:02:09.773 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:02:09.773 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:02:09.773 139328 ERROR oslo_service.periodic_task >2018-06-24 17:02:39.655 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:02:39.656 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:02:39.663 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:02:39.664 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:02:39.696 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:02:39.696 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:02:39.696 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:02:39.696 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:02:39.696 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:02:39.696 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:02:39.696 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:02:39.698 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:02:39.699 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:02:39.699 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:02:49.699 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:02:49.732 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:02:49.732 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:02:49.732 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:02:49.732 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:02:49.732 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:02:49.732 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:02:49.732 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:02:49.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:02:49.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:02:49.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:03:09.736 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:03:09.770 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:03:09.770 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:03:09.770 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:03:09.770 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:03:09.770 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:03:09.770 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:03:09.770 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:03:09.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:03:09.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:03:09.773 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:03:09.773 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:03:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:03:09.773 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:03:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:03:09.773 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:03:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:03:09.773 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:03:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:03:09.773 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:03:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:03:09.773 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:03:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:03:09.773 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:03:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:03:09.773 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:03:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:03:09.773 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:03:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:03:09.773 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:03:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:03:09.773 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:03:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:03:09.773 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:03:09.773 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:03:09.773 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:03:09.773 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:03:09.773 139328 ERROR oslo_service.periodic_task >2018-06-24 17:03:39.654 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:03:39.656 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:03:39.664 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:03:39.665 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:03:39.700 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:03:39.700 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:03:39.700 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:03:39.700 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:03:39.700 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:03:39.700 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:03:39.700 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:03:39.703 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:03:39.703 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:03:39.703 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:03:49.704 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:03:49.737 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:03:49.737 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:03:49.737 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:03:49.737 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:03:49.737 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:03:49.737 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:03:49.737 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:03:49.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:03:49.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:03:49.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:04:09.741 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:04:09.778 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:04:09.778 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:04:09.778 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:04:09.778 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:04:09.778 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:04:09.778 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:04:09.778 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:04:09.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:04:09.781 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:04:09.781 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:04:09.781 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:04:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:04:09.781 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:04:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:04:09.781 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:04:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:04:09.781 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:04:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:04:09.781 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:04:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:04:09.781 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:04:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:04:09.781 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:04:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:04:09.781 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:04:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:04:09.781 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:04:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:04:09.781 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:04:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:04:09.781 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:04:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:04:09.781 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:04:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:04:09.781 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:04:09.781 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:04:09.781 139328 ERROR oslo_service.periodic_task >2018-06-24 17:04:39.652 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:04:39.653 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:04:39.662 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:04:39.662 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:04:39.695 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:04:39.695 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:04:39.695 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:04:39.695 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:04:39.695 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:04:39.695 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:04:39.695 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:04:39.698 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:04:39.698 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:04:39.698 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:04:49.699 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:04:49.741 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:04:49.741 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:04:49.741 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:04:49.741 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:04:49.741 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:04:49.741 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:04:49.741 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:04:49.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:04:49.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.08 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:04:49.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:05:09.744 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:05:09.776 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:05:09.776 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:05:09.776 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:05:09.776 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:05:09.776 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:05:09.776 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:05:09.776 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:05:09.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:05:09.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:05:09.779 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:05:09.779 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:05:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:05:09.779 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:05:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:05:09.779 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:05:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:05:09.779 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:05:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:05:09.779 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:05:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:05:09.779 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:05:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:05:09.779 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:05:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:05:09.779 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:05:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:05:09.779 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:05:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:05:09.779 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:05:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:05:09.779 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:05:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:05:09.779 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:05:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:05:09.779 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:05:09.779 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:05:09.779 139328 ERROR oslo_service.periodic_task >2018-06-24 17:05:39.650 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:05:39.651 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:05:39.659 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:05:39.660 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:05:39.693 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:05:39.693 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:05:39.693 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:05:39.693 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:05:39.693 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:05:39.693 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:05:39.693 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:05:39.696 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:05:39.696 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:05:39.696 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:05:49.697 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:05:49.735 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:05:49.735 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:05:49.735 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:05:49.735 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:05:49.735 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:05:49.735 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:05:49.735 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:05:49.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:05:49.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:05:49.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:06:09.739 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:06:09.772 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:06:09.772 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:06:09.772 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:06:09.772 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:06:09.772 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:06:09.772 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:06:09.772 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:06:09.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:06:09.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:06:09.774 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:06:09.774 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:06:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:06:09.774 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:06:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:06:09.774 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:06:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:06:09.774 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:06:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:06:09.774 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:06:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:06:09.774 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:06:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:06:09.774 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:06:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:06:09.774 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:06:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:06:09.774 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:06:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:06:09.774 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:06:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:06:09.774 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:06:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:06:09.774 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:06:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:06:09.774 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:06:09.774 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:06:09.774 139328 ERROR oslo_service.periodic_task >2018-06-24 17:06:39.646 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:06:39.647 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:06:39.656 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:06:39.657 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:06:39.691 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:06:39.691 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:06:39.691 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:06:39.691 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:06:39.691 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:06:39.691 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:06:39.691 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:06:39.694 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:06:39.695 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:06:39.695 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:06:49.696 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:06:49.732 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:06:49.732 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:06:49.732 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:06:49.732 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:06:49.732 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:06:49.732 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:06:49.732 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:06:49.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:06:49.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:06:49.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:07:09.736 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:07:09.768 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:07:09.768 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:07:09.768 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:07:09.768 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:07:09.768 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:07:09.768 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:07:09.768 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:07:09.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:07:09.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:07:09.770 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:07:09.770 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:07:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:07:09.770 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:07:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:07:09.770 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:07:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:07:09.770 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:07:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:07:09.770 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:07:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:07:09.770 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:07:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:07:09.770 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:07:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:07:09.770 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:07:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:07:09.770 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:07:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:07:09.770 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:07:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:07:09.770 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:07:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:07:09.770 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:07:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:07:09.770 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:07:09.770 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:07:09.770 139328 ERROR oslo_service.periodic_task >2018-06-24 17:07:39.642 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:07:39.643 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:07:39.650 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:07:39.651 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:07:39.684 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:07:39.684 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:07:39.684 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:07:39.684 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:07:39.684 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:07:39.684 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:07:39.684 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:07:39.686 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:07:39.687 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:07:39.687 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:07:49.688 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:07:49.720 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:07:49.720 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:07:49.720 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:07:49.720 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:07:49.720 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:07:49.720 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:07:49.720 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:07:49.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:07:49.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:07:49.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:08:09.723 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:08:09.758 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:08:09.758 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:08:09.758 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:08:09.758 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:08:09.758 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:08:09.758 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:08:09.758 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:08:09.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:08:09.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:08:09.762 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:08:09.762 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:08:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:08:09.762 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:08:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:08:09.762 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:08:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:08:09.762 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:08:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:08:09.762 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:08:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:08:09.762 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:08:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:08:09.762 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:08:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:08:09.762 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:08:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:08:09.762 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:08:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:08:09.762 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:08:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:08:09.762 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:08:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:08:09.762 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:08:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:08:09.762 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:08:09.762 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:08:09.762 139328 ERROR oslo_service.periodic_task >2018-06-24 17:08:39.643 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:08:39.644 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:08:39.653 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:08:39.654 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:08:39.689 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:08:39.689 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:08:39.689 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:08:39.689 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:08:39.689 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:08:39.689 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:08:39.689 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:08:39.692 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:08:39.692 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:08:39.693 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:08:49.693 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:08:49.730 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:08:49.730 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:08:49.730 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:08:49.730 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:08:49.730 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:08:49.730 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:08:49.730 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:08:49.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:08:49.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:08:49.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:09:09.734 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:09:09.767 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:09:09.767 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:09:09.767 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:09:09.767 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:09:09.767 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:09:09.767 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:09:09.767 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:09:09.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:09:09.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:09:09.769 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:09:09.769 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:09:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:09:09.769 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:09:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:09:09.769 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:09:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:09:09.769 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:09:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:09:09.769 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:09:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:09:09.769 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:09:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:09:09.769 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:09:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:09:09.769 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:09:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:09:09.769 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:09:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:09:09.769 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:09:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:09:09.769 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:09:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:09:09.769 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:09:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:09:09.769 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:09:09.769 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:09:09.769 139328 ERROR oslo_service.periodic_task >2018-06-24 17:09:39.640 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:09:39.641 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:09:39.658 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:09:39.658 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:09:39.692 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:09:39.692 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:09:39.692 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:09:39.692 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:09:39.692 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:09:39.692 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:09:39.692 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:09:39.694 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:09:39.694 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:09:39.694 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:09:49.695 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:09:49.730 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:09:49.730 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:09:49.730 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:09:49.730 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:09:49.730 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:09:49.730 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:09:49.730 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:09:49.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:09:49.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:09:49.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:10:09.733 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:10:09.773 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:10:09.773 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:10:09.773 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:10:09.773 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:10:09.773 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:10:09.773 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:10:09.773 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:10:09.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:10:09.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:10:09.775 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:10:09.775 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:10:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:10:09.775 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:10:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:10:09.775 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:10:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:10:09.775 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:10:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:10:09.775 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:10:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:10:09.775 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:10:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:10:09.775 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:10:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:10:09.775 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:10:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:10:09.775 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:10:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:10:09.775 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:10:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:10:09.775 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:10:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:10:09.775 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:10:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:10:09.775 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:10:09.775 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:10:09.775 139328 ERROR oslo_service.periodic_task >2018-06-24 17:10:39.637 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:10:39.638 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:10:39.646 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:10:39.647 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:10:39.688 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:10:39.688 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:10:39.688 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:10:39.688 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:10:39.688 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:10:39.688 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:10:39.688 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:10:39.692 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:10:39.692 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.045 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:10:39.693 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:10:49.694 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:10:49.727 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:10:49.727 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:10:49.727 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:10:49.727 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:10:49.727 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:10:49.727 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:10:49.727 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:10:49.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:10:49.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.083 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:10:49.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:11:09.730 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:11:09.764 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:11:09.764 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:11:09.764 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:11:09.764 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:11:09.764 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:11:09.764 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:11:09.764 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:11:09.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:11:09.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.12 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:11:09.767 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:11:09.767 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:11:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:11:09.767 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:11:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:11:09.767 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:11:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:11:09.767 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:11:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:11:09.767 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:11:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:11:09.767 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:11:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:11:09.767 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:11:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:11:09.767 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:11:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:11:09.767 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:11:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:11:09.767 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:11:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:11:09.767 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:11:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:11:09.767 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:11:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:11:09.767 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:11:09.767 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:11:09.767 139328 ERROR oslo_service.periodic_task >2018-06-24 17:11:39.638 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:11:39.639 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:11:39.647 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:11:39.648 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:11:39.687 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:11:39.687 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:11:39.687 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:11:39.687 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:11:39.687 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:11:39.687 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:11:39.687 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:11:39.689 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:11:39.689 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:11:39.690 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:11:49.690 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:11:49.724 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:11:49.724 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:11:49.724 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:11:49.724 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:11:49.724 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:11:49.724 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:11:49.724 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:11:49.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:11:49.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:11:49.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:12:09.727 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:12:09.760 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:12:09.760 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:12:09.760 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:12:09.760 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:12:09.760 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:12:09.760 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:12:09.760 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:12:09.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:12:09.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:12:09.763 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:12:09.763 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:12:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:12:09.763 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:12:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:12:09.763 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:12:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:12:09.763 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:12:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:12:09.763 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:12:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:12:09.763 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:12:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:12:09.763 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:12:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:12:09.763 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:12:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:12:09.763 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:12:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:12:09.763 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:12:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:12:09.763 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:12:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:12:09.763 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:12:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:12:09.763 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:12:09.763 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:12:09.763 139328 ERROR oslo_service.periodic_task >2018-06-24 17:12:39.634 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:12:39.635 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:12:39.644 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:12:39.644 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:12:39.680 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:12:39.680 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:12:39.680 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:12:39.680 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:12:39.680 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:12:39.680 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:12:39.680 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:12:39.682 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:12:39.682 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:12:39.683 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:12:49.683 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:12:49.718 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:12:49.718 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:12:49.718 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:12:49.718 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:12:49.718 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:12:49.718 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:12:49.718 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:12:49.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:12:49.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:12:49.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:13:09.721 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:13:09.756 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:13:09.756 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:13:09.756 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:13:09.756 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:13:09.756 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:13:09.756 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:13:09.756 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:13:09.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:13:09.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:13:09.758 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:13:09.758 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:13:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:13:09.758 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:13:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:13:09.758 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:13:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:13:09.758 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:13:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:13:09.758 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:13:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:13:09.758 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:13:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:13:09.758 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:13:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:13:09.758 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:13:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:13:09.758 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:13:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:13:09.758 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:13:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:13:09.758 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:13:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:13:09.758 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:13:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:13:09.758 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:13:09.758 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:13:09.758 139328 ERROR oslo_service.periodic_task >2018-06-24 17:13:39.630 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:13:39.631 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:13:39.638 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:13:39.638 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:13:39.671 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:13:39.671 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:13:39.671 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:13:39.671 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:13:39.671 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:13:39.671 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:13:39.671 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:13:39.673 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:13:39.673 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:13:39.673 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:13:49.674 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:13:49.709 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:13:49.709 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:13:49.709 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:13:49.709 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:13:49.709 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:13:49.709 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:13:49.709 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:13:49.711 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:13:49.711 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:13:49.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:14:09.712 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:14:09.745 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:14:09.745 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:14:09.745 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:14:09.745 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:14:09.745 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:14:09.745 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:14:09.745 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:14:09.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:14:09.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:14:09.747 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:14:09.747 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:14:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:14:09.747 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:14:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:14:09.747 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:14:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:14:09.747 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:14:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:14:09.747 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:14:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:14:09.747 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:14:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:14:09.747 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:14:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:14:09.747 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:14:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:14:09.747 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:14:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:14:09.747 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:14:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:14:09.747 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:14:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:14:09.747 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:14:09.747 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:14:09.747 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:14:09.747 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:14:09.747 139328 ERROR oslo_service.periodic_task >2018-06-24 17:14:39.628 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:14:39.629 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:14:39.636 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:14:39.636 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:14:39.671 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:14:39.671 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:14:39.671 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:14:39.671 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:14:39.671 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:14:39.671 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:14:39.671 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:14:39.673 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:14:39.674 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:14:39.674 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:14:49.675 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:14:49.710 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:14:49.710 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:14:49.710 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:14:49.710 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:14:49.710 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:14:49.710 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:14:49.710 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:14:49.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:14:49.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:14:49.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:15:09.715 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:15:09.748 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:15:09.748 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:15:09.748 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:15:09.748 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:15:09.748 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:15:09.748 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:15:09.748 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:15:09.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:15:09.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:15:09.751 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:15:09.751 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:15:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:15:09.751 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:15:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:15:09.751 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:15:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:15:09.751 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:15:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:15:09.751 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:15:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:15:09.751 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:15:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:15:09.751 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:15:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:15:09.751 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:15:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:15:09.751 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:15:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:15:09.751 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:15:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:15:09.751 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:15:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:15:09.751 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:15:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:15:09.751 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:15:09.751 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:15:09.751 139328 ERROR oslo_service.periodic_task >2018-06-24 17:15:39.633 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:15:39.634 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:15:39.641 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:15:39.642 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:15:39.676 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:15:39.676 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:15:39.676 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:15:39.676 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:15:39.676 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:15:39.676 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:15:39.676 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:15:39.678 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:15:39.679 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:15:39.679 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:15:49.680 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:15:49.712 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:15:49.712 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:15:49.712 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:15:49.712 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:15:49.712 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:15:49.712 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:15:49.712 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:15:49.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:15:49.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:15:49.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:16:09.715 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:16:09.749 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:16:09.749 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:16:09.749 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:16:09.749 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:16:09.749 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:16:09.749 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:16:09.749 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:16:09.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:16:09.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:16:09.752 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:16:09.752 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:16:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:16:09.752 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:16:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:16:09.752 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:16:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:16:09.752 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:16:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:16:09.752 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:16:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:16:09.752 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:16:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:16:09.752 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:16:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:16:09.752 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:16:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:16:09.752 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:16:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:16:09.752 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:16:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:16:09.752 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:16:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:16:09.752 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:16:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:16:09.752 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:16:09.752 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:16:09.752 139328 ERROR oslo_service.periodic_task >2018-06-24 17:16:39.634 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:16:39.635 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:16:39.643 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:16:39.644 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:16:39.680 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:16:39.680 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:16:39.680 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:16:39.680 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:16:39.680 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:16:39.680 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:16:39.680 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:16:39.684 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:16:39.684 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:16:39.685 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:16:49.685 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:16:49.719 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:16:49.719 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:16:49.719 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:16:49.719 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:16:49.719 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:16:49.719 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:16:49.719 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:16:49.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:16:49.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:16:49.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:17:09.722 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:17:09.757 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:17:09.757 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:17:09.757 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:17:09.757 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:17:09.757 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:17:09.757 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:17:09.757 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:17:09.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:17:09.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:17:09.760 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:17:09.760 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:17:09.760 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:17:09.760 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:17:09.760 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:17:09.760 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:17:09.760 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:17:09.760 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:17:09.760 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:17:09.760 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:17:09.760 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:17:09.760 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:17:09.760 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:17:09.760 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:17:09.760 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:17:09.760 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:17:09.760 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:17:09.760 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:17:09.760 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:17:09.760 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:17:09.760 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:17:09.760 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:17:09.760 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:17:09.760 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:17:09.760 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:17:09.760 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:17:09.760 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:17:09.760 139328 ERROR oslo_service.periodic_task >2018-06-24 17:17:39.632 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:17:39.632 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:17:39.639 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:17:39.639 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:17:39.673 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:17:39.673 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:17:39.673 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:17:39.673 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:17:39.673 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:17:39.673 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:17:39.673 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:17:39.675 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:17:39.675 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:17:39.676 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:17:49.676 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:17:49.708 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:17:49.708 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:17:49.708 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:17:49.708 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:17:49.708 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:17:49.708 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:17:49.708 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:17:49.710 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:17:49.710 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:17:49.711 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:18:09.711 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:18:09.747 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:18:09.747 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:18:09.747 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:18:09.747 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:18:09.747 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:18:09.747 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:18:09.747 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:18:09.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:18:09.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:18:09.750 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:18:09.750 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:18:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:18:09.750 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:18:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:18:09.750 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:18:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:18:09.750 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:18:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:18:09.750 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:18:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:18:09.750 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:18:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:18:09.750 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:18:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:18:09.750 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:18:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:18:09.750 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:18:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:18:09.750 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:18:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:18:09.750 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:18:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:18:09.750 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:18:09.750 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:18:09.750 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:18:09.750 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:18:09.750 139328 ERROR oslo_service.periodic_task >2018-06-24 17:18:39.632 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:18:39.633 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:18:39.640 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:18:39.641 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:18:39.675 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:18:39.675 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:18:39.675 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:18:39.675 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:18:39.675 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:18:39.675 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:18:39.675 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:18:39.678 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:18:39.678 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:18:39.679 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:18:49.679 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:18:49.726 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:18:49.726 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:18:49.726 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:18:49.726 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:18:49.726 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:18:49.726 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:18:49.726 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:18:49.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:18:49.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.088 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:18:49.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:19:09.729 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:19:09.763 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:19:09.763 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:19:09.763 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:19:09.763 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:19:09.763 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:19:09.763 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:19:09.763 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:19:09.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:19:09.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.125 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:19:09.766 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:19:09.766 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:19:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:19:09.766 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:19:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:19:09.766 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:19:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:19:09.766 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:19:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:19:09.766 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:19:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:19:09.766 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:19:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:19:09.766 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:19:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:19:09.766 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:19:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:19:09.766 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:19:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:19:09.766 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:19:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:19:09.766 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:19:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:19:09.766 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:19:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:19:09.766 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:19:09.766 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:19:09.766 139328 ERROR oslo_service.periodic_task >2018-06-24 17:19:39.627 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:19:39.628 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:19:39.635 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:19:39.636 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:19:39.671 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:19:39.671 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:19:39.671 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:19:39.671 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:19:39.671 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:19:39.671 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:19:39.671 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:19:39.673 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:19:39.673 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:19:39.673 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:19:49.674 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:19:49.715 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:19:49.715 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:19:49.715 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:19:49.715 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:19:49.715 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:19:49.715 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:19:49.715 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:19:49.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:19:49.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.082 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:19:49.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:20:09.719 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:20:09.752 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:20:09.752 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:20:09.752 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:20:09.752 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:20:09.752 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:20:09.752 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:20:09.752 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:20:09.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:20:09.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.119 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:20:09.755 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:20:09.755 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:20:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:20:09.755 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:20:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:20:09.755 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:20:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:20:09.755 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:20:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:20:09.755 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:20:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:20:09.755 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:20:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:20:09.755 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:20:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:20:09.755 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:20:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:20:09.755 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:20:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:20:09.755 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:20:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:20:09.755 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:20:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:20:09.755 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:20:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:20:09.755 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:20:09.755 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:20:09.755 139328 ERROR oslo_service.periodic_task >2018-06-24 17:20:39.627 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:20:39.628 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:20:39.635 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:20:39.636 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:20:39.681 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:20:39.681 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:20:39.681 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:20:39.681 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:20:39.681 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:20:39.681 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:20:39.681 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:20:39.684 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:20:39.685 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.049 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:20:39.685 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:20:49.685 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:20:49.717 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:20:49.717 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:20:49.717 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:20:49.717 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:20:49.717 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:20:49.717 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:20:49.717 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:20:49.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:20:49.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.084 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:20:49.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:21:09.721 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:21:09.754 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:21:09.754 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:21:09.754 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:21:09.754 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:21:09.754 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:21:09.754 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:21:09.754 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:21:09.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:21:09.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.12 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:21:09.757 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:21:09.757 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:21:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:21:09.757 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:21:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:21:09.757 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:21:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:21:09.757 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:21:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:21:09.757 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:21:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:21:09.757 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:21:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:21:09.757 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:21:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:21:09.757 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:21:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:21:09.757 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:21:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:21:09.757 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:21:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:21:09.757 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:21:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:21:09.757 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:21:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:21:09.757 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:21:09.757 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:21:09.757 139328 ERROR oslo_service.periodic_task >2018-06-24 17:21:39.629 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:21:39.630 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:21:39.638 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:21:39.639 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:21:39.675 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:21:39.675 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:21:39.675 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:21:39.675 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:21:39.675 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:21:39.675 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:21:39.675 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:21:39.678 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:21:39.678 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:21:39.679 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:21:49.679 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:21:49.710 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:21:49.710 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:21:49.710 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:21:49.710 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:21:49.710 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:21:49.710 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:21:49.710 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:21:49.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:21:49.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:21:49.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:22:09.713 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:22:09.746 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:22:09.746 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:22:09.746 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:22:09.746 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:22:09.746 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:22:09.746 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:22:09.746 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:22:09.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:22:09.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:22:09.749 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:22:09.749 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:22:09.749 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:22:09.749 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:22:09.749 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:22:09.749 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:22:09.749 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:22:09.749 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:22:09.749 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:22:09.749 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:22:09.749 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:22:09.749 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:22:09.749 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:22:09.749 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:22:09.749 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:22:09.749 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:22:09.749 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:22:09.749 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:22:09.749 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:22:09.749 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:22:09.749 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:22:09.749 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:22:09.749 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:22:09.749 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:22:09.749 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:22:09.749 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:22:09.749 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:22:09.749 139328 ERROR oslo_service.periodic_task >2018-06-24 17:22:39.630 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:22:39.631 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:22:39.638 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:22:39.639 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:22:39.672 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:22:39.672 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:22:39.672 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:22:39.672 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:22:39.672 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:22:39.672 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:22:39.672 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:22:39.674 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:22:39.675 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:22:39.675 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:22:49.675 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:22:49.709 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:22:49.709 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:22:49.709 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:22:49.709 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:22:49.709 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:22:49.709 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:22:49.709 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:22:49.711 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:22:49.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:22:49.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:23:09.712 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:23:09.746 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:23:09.746 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:23:09.746 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:23:09.746 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:23:09.746 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:23:09.746 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:23:09.746 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:23:09.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:23:09.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:23:09.751 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:23:09.751 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:23:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:23:09.751 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:23:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:23:09.751 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:23:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:23:09.751 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:23:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:23:09.751 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:23:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:23:09.751 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:23:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:23:09.751 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:23:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:23:09.751 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:23:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:23:09.751 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:23:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:23:09.751 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:23:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:23:09.751 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:23:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:23:09.751 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:23:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:23:09.751 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:23:09.751 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:23:09.751 139328 ERROR oslo_service.periodic_task >2018-06-24 17:23:39.633 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:23:39.634 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:23:39.643 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:23:39.644 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:23:39.678 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:23:39.678 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:23:39.678 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:23:39.678 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:23:39.678 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:23:39.678 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:23:39.678 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:23:39.681 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:23:39.681 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:23:39.681 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:23:49.682 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:23:49.721 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:23:49.721 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:23:49.721 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:23:49.721 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:23:49.721 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:23:49.721 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:23:49.721 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:23:49.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:23:49.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:23:49.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:24:09.724 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:24:09.759 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:24:09.759 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:24:09.759 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:24:09.759 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:24:09.759 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:24:09.759 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:24:09.759 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:24:09.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:24:09.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:24:09.762 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:24:09.762 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:24:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:24:09.762 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:24:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:24:09.762 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:24:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:24:09.762 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:24:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:24:09.762 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:24:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:24:09.762 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:24:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:24:09.762 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:24:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:24:09.762 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:24:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:24:09.762 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:24:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:24:09.762 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:24:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:24:09.762 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:24:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:24:09.762 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:24:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:24:09.762 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:24:09.762 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:24:09.762 139328 ERROR oslo_service.periodic_task >2018-06-24 17:24:39.634 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:24:39.635 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:25:39.634 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:25:39.635 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:25:39.644 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:25:39.645 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:25:39.684 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:25:39.684 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:25:39.684 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:25:39.684 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:25:39.684 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:25:39.684 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:25:39.684 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:25:39.687 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:25:39.688 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.043 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:25:39.688 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:25:49.689 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:25:49.728 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:25:49.728 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:25:49.728 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:25:49.728 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:25:49.728 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:25:49.728 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:25:49.728 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:25:49.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:25:49.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.086 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:25:49.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:26:09.732 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:26:09.765 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:26:09.765 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:26:09.765 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:26:09.765 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:26:09.765 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:26:09.765 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:26:09.765 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:26:09.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:26:09.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.123 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:26:09.768 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:26:09.768 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:26:09.768 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:26:09.768 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:26:09.768 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:26:09.768 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:26:09.768 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:26:09.768 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:26:09.768 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:26:09.768 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:26:09.768 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:26:09.768 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:26:09.768 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:26:09.768 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:26:09.768 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:26:09.768 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:26:09.768 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:26:09.768 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:26:09.768 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:26:09.768 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:26:09.768 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:26:09.768 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:26:09.768 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:26:09.768 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:26:09.768 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:26:09.768 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:26:09.768 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:26:09.768 139328 ERROR oslo_service.periodic_task >2018-06-24 17:26:39.629 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:26:39.631 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:26:39.639 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:26:39.640 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:26:39.674 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:26:39.674 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:26:39.674 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:26:39.674 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:26:39.674 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:26:39.674 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:26:39.674 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:26:39.676 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:26:39.677 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:26:39.677 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:26:49.677 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:26:49.719 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:26:49.719 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:26:49.719 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:26:49.719 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:26:49.719 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:26:49.719 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:26:49.719 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:26:49.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:26:49.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.081 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:26:49.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:27:09.722 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:27:09.756 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:27:09.756 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:27:09.756 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:27:09.756 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:27:09.756 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:27:09.756 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:27:09.756 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:27:09.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:27:09.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.119 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:27:09.759 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:27:09.759 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:27:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:27:09.759 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:27:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:27:09.759 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:27:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:27:09.759 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:27:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:27:09.759 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:27:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:27:09.759 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:27:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:27:09.759 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:27:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:27:09.759 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:27:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:27:09.759 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:27:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:27:09.759 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:27:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:27:09.759 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:27:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:27:09.759 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:27:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:27:09.759 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:27:09.759 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:27:09.759 139328 ERROR oslo_service.periodic_task >2018-06-24 17:27:39.630 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:27:39.631 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:27:39.638 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:27:39.639 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:27:39.674 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:27:39.674 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:27:39.674 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:27:39.674 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:27:39.674 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:27:39.674 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:27:39.674 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:27:39.677 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:27:39.677 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:27:39.677 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:27:49.678 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:27:49.711 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:27:49.711 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:27:49.711 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:27:49.711 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:27:49.711 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:27:49.711 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:27:49.711 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:27:49.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:27:49.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:27:49.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:28:09.714 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:28:09.750 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:28:09.750 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:28:09.750 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:28:09.750 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:28:09.750 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:28:09.750 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:28:09.750 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:28:09.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:28:09.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:28:09.753 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:28:09.753 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:28:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:28:09.753 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:28:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:28:09.753 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:28:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:28:09.753 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:28:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:28:09.753 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:28:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:28:09.753 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:28:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:28:09.753 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:28:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:28:09.753 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:28:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:28:09.753 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:28:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:28:09.753 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:28:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:28:09.753 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:28:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:28:09.753 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:28:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:28:09.753 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:28:09.753 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:28:09.753 139328 ERROR oslo_service.periodic_task >2018-06-24 17:28:39.634 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:28:39.635 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:28:39.643 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:28:39.644 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:28:39.677 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:28:39.677 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:28:39.677 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:28:39.677 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:28:39.677 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:28:39.677 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:28:39.677 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:28:39.679 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:28:39.680 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:28:39.680 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:28:49.680 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:28:49.729 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:28:49.729 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:28:49.729 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:28:49.729 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:28:49.729 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:28:49.729 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:28:49.729 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:28:49.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:28:49.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.088 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:28:49.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:29:09.733 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:29:09.766 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:29:09.766 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:29:09.766 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:29:09.766 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:29:09.766 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:29:09.766 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:29:09.766 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:29:09.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:29:09.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.125 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:29:09.769 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:29:09.769 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:29:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:29:09.769 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:29:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:29:09.769 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:29:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:29:09.769 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:29:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:29:09.769 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:29:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:29:09.769 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:29:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:29:09.769 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:29:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:29:09.769 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:29:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:29:09.769 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:29:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:29:09.769 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:29:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:29:09.769 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:29:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:29:09.769 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:29:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:29:09.769 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:29:09.769 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:29:09.769 139328 ERROR oslo_service.periodic_task >2018-06-24 17:29:39.631 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:29:39.632 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:29:39.639 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:29:39.639 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:29:39.673 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:29:39.673 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:29:39.673 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:29:39.673 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:29:39.673 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:29:39.673 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:29:39.673 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:29:39.676 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:29:39.676 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:29:39.676 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:29:49.677 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:29:49.714 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:29:49.714 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:29:49.714 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:29:49.714 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:29:49.714 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:29:49.714 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:29:49.714 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:29:49.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:29:49.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:29:49.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:30:09.718 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:30:09.754 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:30:09.754 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:30:09.754 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:30:09.754 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:30:09.754 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:30:09.754 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:30:09.754 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:30:09.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:30:09.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:30:09.757 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:30:09.757 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:30:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:30:09.757 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:30:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:30:09.757 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:30:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:30:09.757 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:30:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:30:09.757 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:30:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:30:09.757 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:30:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:30:09.757 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:30:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:30:09.757 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:30:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:30:09.757 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:30:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:30:09.757 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:30:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:30:09.757 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:30:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:30:09.757 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:30:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:30:09.757 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:30:09.757 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:30:09.757 139328 ERROR oslo_service.periodic_task >2018-06-24 17:30:39.628 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:30:39.629 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:30:39.636 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:30:39.637 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:30:39.670 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:30:39.670 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:30:39.670 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:30:39.670 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:30:39.670 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:30:39.670 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:30:39.670 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:30:39.672 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:30:39.673 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:30:39.673 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:30:49.673 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:30:49.709 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:30:49.709 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:30:49.709 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:30:49.709 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:30:49.709 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:30:49.709 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:30:49.709 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:30:49.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:30:49.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:30:49.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:31:09.713 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:31:09.747 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:31:09.747 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:31:09.747 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:31:09.747 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:31:09.747 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:31:09.747 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:31:09.747 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:31:09.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:31:09.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:31:09.749 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:31:09.749 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:31:09.749 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:31:09.749 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:31:09.749 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:31:09.749 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:31:09.749 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:31:09.749 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:31:09.749 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:31:09.749 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:31:09.749 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:31:09.749 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:31:09.749 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:31:09.749 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:31:09.749 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:31:09.749 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:31:09.749 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:31:09.749 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:31:09.749 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:31:09.749 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:31:09.749 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:31:09.749 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:31:09.749 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:31:09.749 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:31:09.749 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:31:09.749 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:31:09.749 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:31:09.749 139328 ERROR oslo_service.periodic_task >2018-06-24 17:31:39.631 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:31:39.632 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:31:39.639 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:31:39.640 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:31:39.676 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:31:39.676 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:31:39.676 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:31:39.676 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:31:39.676 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:31:39.676 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:31:39.676 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:31:39.678 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:31:39.678 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:31:39.679 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:31:49.679 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:31:49.712 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:31:49.712 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:31:49.712 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:31:49.712 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:31:49.712 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:31:49.712 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:31:49.712 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:31:49.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:31:49.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:31:49.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:32:09.715 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:32:09.749 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:32:09.749 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:32:09.749 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:32:09.749 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:32:09.749 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:32:09.749 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:32:09.749 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:32:09.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:32:09.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:32:09.752 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:32:09.752 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:32:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:32:09.752 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:32:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:32:09.752 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:32:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:32:09.752 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:32:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:32:09.752 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:32:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:32:09.752 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:32:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:32:09.752 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:32:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:32:09.752 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:32:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:32:09.752 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:32:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:32:09.752 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:32:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:32:09.752 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:32:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:32:09.752 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:32:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:32:09.752 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:32:09.752 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:32:09.752 139328 ERROR oslo_service.periodic_task >2018-06-24 17:32:39.633 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:32:39.635 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:32:39.643 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:32:39.644 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:32:39.680 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:32:39.680 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:32:39.680 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:32:39.680 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:32:39.680 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:32:39.680 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:32:39.680 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:32:39.683 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:32:39.683 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:32:39.683 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:32:49.684 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:32:49.716 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:32:49.716 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:32:49.716 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:32:49.716 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:32:49.716 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:32:49.716 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:32:49.716 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:32:49.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:32:49.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:32:49.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:33:09.720 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:33:09.753 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:33:09.753 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:33:09.753 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:33:09.753 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:33:09.753 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:33:09.753 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:33:09.753 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:33:09.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:33:09.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:33:09.756 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:33:09.756 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:33:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:33:09.756 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:33:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:33:09.756 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:33:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:33:09.756 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:33:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:33:09.756 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:33:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:33:09.756 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:33:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:33:09.756 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:33:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:33:09.756 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:33:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:33:09.756 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:33:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:33:09.756 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:33:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:33:09.756 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:33:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:33:09.756 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:33:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:33:09.756 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:33:09.756 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:33:09.756 139328 ERROR oslo_service.periodic_task >2018-06-24 17:33:39.637 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:33:39.638 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:33:39.645 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:33:39.645 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:33:39.678 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:33:39.678 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:33:39.678 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:33:39.678 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:33:39.678 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:33:39.678 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:33:39.678 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:33:39.679 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:33:39.680 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:33:39.680 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:33:49.681 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:33:49.713 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:33:49.713 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:33:49.713 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:33:49.713 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:33:49.713 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:33:49.713 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:33:49.713 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:33:49.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:33:49.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.07 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:33:49.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:34:09.717 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:34:09.749 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:34:09.749 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:34:09.749 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:34:09.749 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:34:09.749 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:34:09.749 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:34:09.749 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:34:09.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:34:09.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.106 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:34:09.752 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:34:09.752 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:34:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:34:09.752 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:34:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:34:09.752 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:34:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:34:09.752 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:34:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:34:09.752 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:34:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:34:09.752 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:34:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:34:09.752 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:34:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:34:09.752 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:34:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:34:09.752 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:34:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:34:09.752 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:34:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:34:09.752 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:34:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:34:09.752 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:34:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:34:09.752 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:34:09.752 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:34:09.752 139328 ERROR oslo_service.periodic_task >2018-06-24 17:34:39.633 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:34:39.634 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:34:39.644 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:34:39.645 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:34:39.687 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:34:39.687 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:34:39.687 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:34:39.687 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:34:39.687 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:34:39.687 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:34:39.687 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:34:39.690 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:34:39.690 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.045 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:34:39.690 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:34:49.691 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:34:49.723 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:34:49.723 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:34:49.723 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:34:49.723 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:34:49.723 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:34:49.723 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:34:49.723 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:34:49.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:34:49.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.081 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:34:49.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:35:09.727 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:35:09.759 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:35:09.759 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:35:09.759 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:35:09.759 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:35:09.759 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:35:09.759 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:35:09.759 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:35:09.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:35:09.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:35:09.762 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:35:09.762 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:35:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:35:09.762 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:35:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:35:09.762 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:35:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:35:09.762 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:35:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:35:09.762 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:35:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:35:09.762 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:35:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:35:09.762 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:35:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:35:09.762 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:35:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:35:09.762 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:35:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:35:09.762 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:35:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:35:09.762 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:35:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:35:09.762 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:35:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:35:09.762 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:35:09.762 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:35:09.762 139328 ERROR oslo_service.periodic_task >2018-06-24 17:35:39.634 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:35:39.635 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:35:39.644 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:35:39.645 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:35:39.679 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:35:39.679 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:35:39.679 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:35:39.679 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:35:39.679 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:35:39.679 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:35:39.679 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:35:39.682 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:35:39.682 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:35:39.682 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:35:49.683 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:35:49.719 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:35:49.719 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:35:49.719 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:35:49.719 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:35:49.719 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:35:49.719 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:35:49.719 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:35:49.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:35:49.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:35:49.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:36:09.723 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:36:09.757 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:36:09.757 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:36:09.757 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:36:09.757 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:36:09.757 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:36:09.757 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:36:09.757 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:36:09.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:36:09.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:36:09.759 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:36:09.759 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:36:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:36:09.759 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:36:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:36:09.759 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:36:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:36:09.759 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:36:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:36:09.759 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:36:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:36:09.759 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:36:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:36:09.759 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:36:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:36:09.759 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:36:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:36:09.759 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:36:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:36:09.759 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:36:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:36:09.759 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:36:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:36:09.759 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:36:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:36:09.759 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:36:09.759 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:36:09.759 139328 ERROR oslo_service.periodic_task >2018-06-24 17:36:39.631 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:36:39.632 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:36:39.638 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:36:39.639 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:36:39.674 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:36:39.674 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:36:39.674 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:36:39.674 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:36:39.674 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:36:39.674 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:36:39.674 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:36:39.677 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:36:39.677 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:36:39.678 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:36:49.678 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:36:49.711 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:36:49.711 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:36:49.711 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:36:49.711 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:36:49.711 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:36:49.711 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:36:49.711 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:36:49.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:36:49.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:36:49.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:37:09.714 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:37:09.755 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:37:09.755 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:37:09.755 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:37:09.755 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:37:09.755 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:37:09.755 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:37:09.755 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:37:09.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:37:09.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.118 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:37:09.758 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:37:09.758 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:37:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:37:09.758 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:37:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:37:09.758 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:37:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:37:09.758 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:37:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:37:09.758 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:37:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:37:09.758 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:37:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:37:09.758 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:37:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:37:09.758 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:37:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:37:09.758 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:37:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:37:09.758 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:37:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:37:09.758 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:37:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:37:09.758 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:37:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:37:09.758 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:37:09.758 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:37:09.758 139328 ERROR oslo_service.periodic_task >2018-06-24 17:37:39.629 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:37:39.630 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:37:39.638 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:37:39.639 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:37:39.675 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:37:39.675 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:37:39.675 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:37:39.675 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:37:39.675 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:37:39.675 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:37:39.675 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:37:39.678 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:37:39.678 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:37:39.678 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:37:49.679 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:37:49.711 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:37:49.711 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:37:49.711 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:37:49.711 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:37:49.711 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:37:49.711 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:37:49.711 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:37:49.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:37:49.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:37:49.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:38:09.714 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:38:09.745 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:38:09.745 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:38:09.745 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:38:09.745 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:38:09.745 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:38:09.745 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:38:09.745 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:38:09.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:38:09.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:38:09.748 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:38:09.748 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:38:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:38:09.748 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:38:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:38:09.748 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:38:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:38:09.748 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:38:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:38:09.748 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:38:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:38:09.748 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:38:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:38:09.748 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:38:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:38:09.748 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:38:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:38:09.748 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:38:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:38:09.748 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:38:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:38:09.748 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:38:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:38:09.748 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:38:09.748 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:38:09.748 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:38:09.748 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:38:09.748 139328 ERROR oslo_service.periodic_task >2018-06-24 17:38:39.629 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:38:39.630 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:38:39.637 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:38:39.637 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:38:39.670 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:38:39.670 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:38:39.670 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:38:39.670 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:38:39.670 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:38:39.670 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:38:39.670 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:38:39.673 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:38:39.673 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:38:39.674 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:38:49.674 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:38:49.708 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:38:49.708 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:38:49.708 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:38:49.708 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:38:49.708 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:38:49.708 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:38:49.708 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:38:49.710 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:38:49.710 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:38:49.711 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:39:09.711 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:39:09.743 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:39:09.743 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:39:09.743 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:39:09.743 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:39:09.743 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:39:09.743 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:39:09.743 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:39:09.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:39:09.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:39:09.746 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:39:09.746 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:39:09.746 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:39:09.746 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:39:09.746 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:39:09.746 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:39:09.746 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:39:09.746 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:39:09.746 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:39:09.746 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:39:09.746 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:39:09.746 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:39:09.746 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:39:09.746 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:39:09.746 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:39:09.746 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:39:09.746 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:39:09.746 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:39:09.746 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:39:09.746 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:39:09.746 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:39:09.746 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:39:09.746 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:39:09.746 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:39:09.746 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:39:09.746 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:39:09.746 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:39:09.746 139328 ERROR oslo_service.periodic_task >2018-06-24 17:39:39.627 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:39:39.628 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:39:39.635 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:39:39.635 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:39:39.669 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:39:39.669 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:39:39.669 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:39:39.669 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:39:39.669 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:39:39.669 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:39:39.669 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:39:39.672 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:39:39.672 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:39:39.672 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:39:49.673 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:39:49.705 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:39:49.705 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:39:49.705 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:39:49.705 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:39:49.705 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:39:49.705 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:39:49.705 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:39:49.708 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:39:49.708 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:39:49.709 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:40:09.710 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:40:09.742 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:40:09.742 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:40:09.742 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:40:09.742 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:40:09.742 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:40:09.742 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:40:09.742 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:40:09.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:40:09.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:40:09.745 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:40:09.745 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:40:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:40:09.745 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:40:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:40:09.745 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:40:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:40:09.745 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:40:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:40:09.745 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:40:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:40:09.745 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:40:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:40:09.745 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:40:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:40:09.745 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:40:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:40:09.745 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:40:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:40:09.745 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:40:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:40:09.745 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:40:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:40:09.745 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:40:09.745 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:40:09.745 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:40:09.745 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:40:09.745 139328 ERROR oslo_service.periodic_task >2018-06-24 17:40:39.626 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:40:39.628 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:40:39.645 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:40:39.645 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:40:39.682 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:40:39.682 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:40:39.682 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:40:39.682 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:40:39.682 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:40:39.682 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:40:39.682 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:40:39.685 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:40:39.685 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:40:39.686 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:40:49.686 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:40:49.720 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:40:49.720 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:40:49.720 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:40:49.720 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:40:49.720 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:40:49.720 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:40:49.720 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:40:49.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:40:49.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:40:49.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:41:09.723 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:41:09.755 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:41:09.755 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:41:09.755 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:41:09.755 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:41:09.755 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:41:09.755 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:41:09.755 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:41:09.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:41:09.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:41:09.758 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:41:09.758 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:41:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:41:09.758 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:41:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:41:09.758 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:41:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:41:09.758 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:41:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:41:09.758 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:41:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:41:09.758 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:41:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:41:09.758 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:41:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:41:09.758 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:41:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:41:09.758 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:41:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:41:09.758 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:41:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:41:09.758 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:41:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:41:09.758 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:41:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:41:09.758 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:41:09.758 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:41:09.758 139328 ERROR oslo_service.periodic_task >2018-06-24 17:41:39.629 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:41:39.630 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:41:39.638 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:41:39.639 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:41:39.687 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:41:39.687 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:41:39.687 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:41:39.687 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:41:39.687 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:41:39.687 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:41:39.687 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:41:39.689 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:41:39.689 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.05 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:41:39.690 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:41:49.690 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:41:49.723 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:41:49.723 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:41:49.723 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:41:49.723 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:41:49.723 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:41:49.723 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:41:49.723 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:41:49.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:41:49.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.087 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:41:49.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:42:09.726 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:42:09.759 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:42:09.759 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:42:09.759 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:42:09.759 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:42:09.759 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:42:09.759 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:42:09.759 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:42:09.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:42:09.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.123 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:42:09.762 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:42:09.762 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:42:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:42:09.762 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:42:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:42:09.762 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:42:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:42:09.762 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:42:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:42:09.762 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:42:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:42:09.762 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:42:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:42:09.762 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:42:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:42:09.762 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:42:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:42:09.762 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:42:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:42:09.762 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:42:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:42:09.762 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:42:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:42:09.762 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:42:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:42:09.762 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:42:09.762 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:42:09.762 139328 ERROR oslo_service.periodic_task >2018-06-24 17:42:39.634 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:42:39.635 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:42:39.642 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:42:39.642 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:42:39.679 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:42:39.679 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:42:39.679 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:42:39.679 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:42:39.679 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:42:39.679 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:42:39.679 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:42:39.682 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:42:39.682 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:42:39.682 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:42:49.683 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:42:49.714 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:42:49.714 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:42:49.714 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:42:49.714 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:42:49.714 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:42:49.714 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:42:49.714 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:42:49.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:42:49.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:42:49.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:43:09.717 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:43:09.750 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:43:09.750 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:43:09.750 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:43:09.750 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:43:09.750 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:43:09.750 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:43:09.750 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:43:09.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:43:09.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:43:09.752 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:43:09.752 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:43:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:43:09.752 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:43:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:43:09.752 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:43:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:43:09.752 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:43:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:43:09.752 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:43:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:43:09.752 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:43:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:43:09.752 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:43:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:43:09.752 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:43:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:43:09.752 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:43:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:43:09.752 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:43:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:43:09.752 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:43:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:43:09.752 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:43:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:43:09.752 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:43:09.752 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:43:09.752 139328 ERROR oslo_service.periodic_task >2018-06-24 17:43:39.634 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:43:39.634 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:43:39.644 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:43:39.645 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:43:39.680 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:43:39.680 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:43:39.680 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:43:39.680 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:43:39.680 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:43:39.680 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:43:39.680 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:43:39.682 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:43:39.683 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:43:39.683 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:43:49.684 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:43:49.725 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:43:49.725 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:43:49.725 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:43:49.725 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:43:49.725 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:43:49.725 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:43:49.725 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:43:49.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:43:49.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.083 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:43:49.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:44:09.731 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:44:09.767 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:44:09.767 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:44:09.767 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:44:09.767 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:44:09.767 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:44:09.767 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:44:09.767 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:44:09.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:44:09.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.125 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:44:09.771 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:44:09.771 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:44:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:44:09.771 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:44:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:44:09.771 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:44:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:44:09.771 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:44:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:44:09.771 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:44:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:44:09.771 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:44:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:44:09.771 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:44:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:44:09.771 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:44:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:44:09.771 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:44:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:44:09.771 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:44:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:44:09.771 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:44:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:44:09.771 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:44:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:44:09.771 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:44:09.771 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:44:09.771 139328 ERROR oslo_service.periodic_task >2018-06-24 17:44:39.633 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:44:39.634 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:44:39.642 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:44:39.642 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:44:39.678 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:44:39.678 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:44:39.678 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:44:39.678 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:44:39.678 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:44:39.678 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:44:39.678 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:44:39.681 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:44:39.681 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:44:39.681 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:44:49.682 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:44:49.715 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:44:49.715 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:44:49.715 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:44:49.715 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:44:49.715 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:44:49.715 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:44:49.715 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:44:49.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:44:49.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:44:49.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:45:09.718 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:45:09.752 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:45:09.752 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:45:09.752 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:45:09.752 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:45:09.752 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:45:09.752 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:45:09.752 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:45:09.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:45:09.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:45:09.755 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:45:09.755 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:45:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:45:09.755 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:45:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:45:09.755 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:45:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:45:09.755 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:45:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:45:09.755 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:45:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:45:09.755 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:45:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:45:09.755 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:45:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:45:09.755 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:45:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:45:09.755 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:45:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:45:09.755 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:45:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:45:09.755 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:45:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:45:09.755 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:45:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:45:09.755 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:45:09.755 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:45:09.755 139328 ERROR oslo_service.periodic_task >2018-06-24 17:45:39.637 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:45:39.638 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:45:39.648 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:45:39.649 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:45:39.687 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:45:39.687 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:45:39.687 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:45:39.687 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:45:39.687 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:45:39.687 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:45:39.687 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:45:39.689 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:45:39.689 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:45:39.690 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:45:49.690 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:45:49.730 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:45:49.730 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:45:49.730 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:45:49.730 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:45:49.730 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:45:49.730 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:45:49.730 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:45:49.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:45:49.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.084 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:45:49.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:46:09.733 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:46:09.766 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:46:09.766 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:46:09.766 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:46:09.766 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:46:09.766 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:46:09.766 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:46:09.766 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:46:09.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:46:09.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.12 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:46:09.769 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:46:09.769 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:46:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:46:09.769 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:46:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:46:09.769 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:46:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:46:09.769 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:46:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:46:09.769 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:46:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:46:09.769 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:46:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:46:09.769 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:46:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:46:09.769 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:46:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:46:09.769 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:46:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:46:09.769 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:46:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:46:09.769 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:46:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:46:09.769 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:46:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:46:09.769 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:46:09.769 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:46:09.769 139328 ERROR oslo_service.periodic_task >2018-06-24 17:46:39.640 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:46:39.642 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:46:39.650 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:46:39.650 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:46:39.685 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:46:39.685 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:46:39.685 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:46:39.685 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:46:39.685 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:46:39.685 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:46:39.685 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:46:39.687 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:46:39.687 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:46:39.688 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:46:49.688 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:46:49.722 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:46:49.722 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:46:49.722 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:46:49.722 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:46:49.722 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:46:49.722 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:46:49.722 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:46:49.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:46:49.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:46:49.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:47:09.725 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:47:09.758 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:47:09.758 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:47:09.758 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:47:09.758 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:47:09.758 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:47:09.758 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:47:09.758 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:47:09.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:47:09.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:47:09.761 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:47:09.761 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:47:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:47:09.761 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:47:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:47:09.761 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:47:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:47:09.761 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:47:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:47:09.761 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:47:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:47:09.761 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:47:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:47:09.761 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:47:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:47:09.761 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:47:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:47:09.761 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:47:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:47:09.761 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:47:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:47:09.761 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:47:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:47:09.761 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:47:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:47:09.761 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:47:09.761 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:47:09.761 139328 ERROR oslo_service.periodic_task >2018-06-24 17:47:39.642 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:47:39.643 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:47:39.653 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:47:39.654 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:47:39.689 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:47:39.689 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:47:39.689 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:47:39.689 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:47:39.689 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:47:39.689 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:47:39.689 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:47:39.692 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:47:39.692 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:47:39.692 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:47:49.693 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:47:49.725 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:47:49.725 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:47:49.725 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:47:49.725 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:47:49.725 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:47:49.725 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:47:49.725 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:47:49.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:47:49.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:47:49.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:48:09.729 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:48:09.765 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:48:09.765 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:48:09.765 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:48:09.765 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:48:09.765 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:48:09.765 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:48:09.765 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:48:09.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:48:09.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:48:09.768 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:48:09.768 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:48:09.768 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:48:09.768 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:48:09.768 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:48:09.768 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:48:09.768 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:48:09.768 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:48:09.768 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:48:09.768 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:48:09.768 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:48:09.768 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:48:09.768 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:48:09.768 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:48:09.768 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:48:09.768 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:48:09.768 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:48:09.768 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:48:09.768 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:48:09.768 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:48:09.768 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:48:09.768 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:48:09.768 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:48:09.768 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:48:09.768 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:48:09.768 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:48:09.768 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:48:09.768 139328 ERROR oslo_service.periodic_task >2018-06-24 17:48:39.639 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:48:39.640 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:48:39.649 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:48:39.649 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:48:39.684 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:48:39.684 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:48:39.684 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:48:39.684 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:48:39.684 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:48:39.684 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:48:39.684 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:48:39.686 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:48:39.686 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:48:39.686 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:48:49.687 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:48:49.720 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:48:49.720 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:48:49.720 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:48:49.720 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:48:49.720 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:48:49.720 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:48:49.720 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:48:49.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:48:49.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:48:49.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:49:09.723 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:49:09.756 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:49:09.756 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:49:09.756 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:49:09.756 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:49:09.756 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:49:09.756 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:49:09.756 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:49:09.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:49:09.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:49:09.758 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:49:09.758 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:49:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:49:09.758 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:49:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:49:09.758 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:49:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:49:09.758 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:49:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:49:09.758 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:49:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:49:09.758 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:49:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:49:09.758 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:49:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:49:09.758 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:49:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:49:09.758 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:49:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:49:09.758 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:49:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:49:09.758 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:49:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:49:09.758 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:49:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:49:09.758 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:49:09.758 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:49:09.758 139328 ERROR oslo_service.periodic_task >2018-06-24 17:49:39.640 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:49:39.641 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:49:39.651 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:49:39.652 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:49:39.687 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:49:39.687 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:49:39.687 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:49:39.687 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:49:39.687 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:49:39.687 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:49:39.687 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:49:39.689 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:49:39.689 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:49:39.690 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:49:49.690 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:49:49.722 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:49:49.722 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:49:49.722 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:49:49.722 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:49:49.722 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:49:49.722 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:49:49.722 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:49:49.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:49:49.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:49:49.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:50:09.725 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:50:09.768 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:50:09.768 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:50:09.768 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:50:09.768 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:50:09.768 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:50:09.768 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:50:09.768 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:50:09.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:50:09.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.119 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:50:09.771 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:50:09.771 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:50:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:50:09.771 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:50:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:50:09.771 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:50:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:50:09.771 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:50:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:50:09.771 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:50:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:50:09.771 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:50:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:50:09.771 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:50:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:50:09.771 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:50:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:50:09.771 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:50:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:50:09.771 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:50:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:50:09.771 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:50:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:50:09.771 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:50:09.771 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:50:09.771 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:50:09.771 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:50:09.771 139328 ERROR oslo_service.periodic_task >2018-06-24 17:50:39.642 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:50:39.643 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:50:39.650 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:50:39.651 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:50:39.685 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:50:39.685 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:50:39.685 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:50:39.685 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:50:39.685 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:50:39.685 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:50:39.685 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:50:39.687 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:50:39.687 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:50:39.688 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:50:49.688 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:50:49.722 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:50:49.722 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:50:49.722 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:50:49.722 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:50:49.722 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:50:49.722 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:50:49.722 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:50:49.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:50:49.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:50:49.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:51:09.725 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:51:09.762 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:51:09.762 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:51:09.762 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:51:09.762 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:51:09.762 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:51:09.762 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:51:09.762 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:51:09.764 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:51:09.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:51:09.765 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:51:09.765 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:51:09.765 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:51:09.765 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:51:09.765 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:51:09.765 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:51:09.765 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:51:09.765 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:51:09.765 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:51:09.765 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:51:09.765 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:51:09.765 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:51:09.765 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:51:09.765 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:51:09.765 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:51:09.765 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:51:09.765 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:51:09.765 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:51:09.765 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:51:09.765 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:51:09.765 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:51:09.765 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:51:09.765 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:51:09.765 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:51:09.765 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:51:09.765 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:51:09.765 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:51:09.765 139328 ERROR oslo_service.periodic_task >2018-06-24 17:51:39.646 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:51:39.648 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:51:39.654 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:51:39.654 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:51:39.687 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:51:39.687 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:51:39.687 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:51:39.687 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:51:39.687 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:51:39.687 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:51:39.687 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:51:39.689 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:51:39.690 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:51:39.690 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:51:49.691 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:51:49.723 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:51:49.723 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:51:49.723 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:51:49.723 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:51:49.723 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:51:49.723 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:51:49.723 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:51:49.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:51:49.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.07 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:51:49.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:52:09.726 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:52:09.763 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:52:09.763 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:52:09.763 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:52:09.763 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:52:09.763 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:52:09.763 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:52:09.763 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:52:09.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:52:09.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:52:09.766 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:52:09.766 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:52:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:52:09.766 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:52:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:52:09.766 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:52:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:52:09.766 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:52:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:52:09.766 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:52:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:52:09.766 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:52:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:52:09.766 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:52:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:52:09.766 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:52:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:52:09.766 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:52:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:52:09.766 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:52:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:52:09.766 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:52:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:52:09.766 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:52:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:52:09.766 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:52:09.766 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:52:09.766 139328 ERROR oslo_service.periodic_task >2018-06-24 17:52:39.648 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:52:39.649 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:52:39.688 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:52:39.688 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:52:39.688 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:52:39.688 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:52:39.688 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:52:39.688 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:52:39.688 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:52:39.691 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:52:39.691 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.042 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:52:39.692 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:52:49.693 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:52:49.728 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:52:49.728 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:52:49.728 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:52:49.728 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:52:49.728 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:52:49.728 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:52:49.728 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:52:49.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:52:49.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.081 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:52:49.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:53:09.731 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:53:09.764 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:53:09.764 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:53:09.764 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:53:09.764 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:53:09.764 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:53:09.764 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:53:09.764 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:53:09.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:53:09.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:53:09.767 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:53:09.767 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:53:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:53:09.767 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:53:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:53:09.767 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:53:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:53:09.767 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:53:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:53:09.767 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:53:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:53:09.767 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:53:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:53:09.767 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:53:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:53:09.767 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:53:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:53:09.767 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:53:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:53:09.767 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:53:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:53:09.767 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:53:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:53:09.767 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:53:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:53:09.767 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:53:09.767 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:53:09.767 139328 ERROR oslo_service.periodic_task >2018-06-24 17:53:39.648 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:53:39.649 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:53:39.657 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:53:39.657 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:53:39.692 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:53:39.692 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:53:39.692 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:53:39.692 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:53:39.692 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:53:39.692 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:53:39.692 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:53:39.695 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:53:39.695 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:53:39.696 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:53:49.696 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:53:49.735 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:53:49.735 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:53:49.735 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:53:49.735 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:53:49.735 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:53:49.735 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:53:49.735 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:53:49.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:53:49.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:53:49.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:54:09.739 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:54:09.772 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:54:09.772 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:54:09.772 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:54:09.772 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:54:09.772 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:54:09.772 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:54:09.772 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:54:09.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:54:09.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.118 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:54:09.776 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:54:09.776 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:54:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:54:09.776 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:54:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:54:09.776 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:54:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:54:09.776 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:54:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:54:09.776 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:54:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:54:09.776 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:54:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:54:09.776 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:54:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:54:09.776 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:54:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:54:09.776 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:54:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:54:09.776 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:54:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:54:09.776 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:54:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:54:09.776 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:54:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:54:09.776 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:54:09.776 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:54:09.776 139328 ERROR oslo_service.periodic_task >2018-06-24 17:54:39.648 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:54:39.649 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:54:39.655 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:54:39.656 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:54:39.688 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:54:39.688 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:54:39.688 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:54:39.688 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:54:39.688 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:54:39.688 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:54:39.688 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:54:39.690 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:54:39.691 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:54:39.691 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:54:49.691 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:54:49.726 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:54:49.726 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:54:49.726 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:54:49.726 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:54:49.726 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:54:49.726 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:54:49.726 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:54:49.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:54:49.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:54:49.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:55:09.729 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:55:09.761 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:55:09.761 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:55:09.761 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:55:09.761 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:55:09.761 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:55:09.761 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:55:09.761 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:55:09.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:55:09.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:55:09.764 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:55:09.764 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:55:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:55:09.764 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:55:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:55:09.764 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:55:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:55:09.764 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:55:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:55:09.764 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:55:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:55:09.764 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:55:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:55:09.764 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:55:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:55:09.764 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:55:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:55:09.764 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:55:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:55:09.764 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:55:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:55:09.764 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:55:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:55:09.764 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:55:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:55:09.764 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:55:09.764 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:55:09.764 139328 ERROR oslo_service.periodic_task >2018-06-24 17:55:39.645 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:55:39.646 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:55:39.653 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:55:39.653 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:55:39.698 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:55:39.698 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:55:39.698 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:55:39.698 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:55:39.698 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:55:39.698 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:55:39.698 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:55:39.700 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:55:39.700 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.046 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:55:39.701 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:55:49.701 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:55:49.737 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:55:49.737 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:55:49.737 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:55:49.737 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:55:49.737 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:55:49.737 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:55:49.737 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:55:49.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:55:49.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.086 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:55:49.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:56:09.741 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:56:09.775 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:56:09.775 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:56:09.775 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:56:09.775 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:56:09.775 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:56:09.775 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:56:09.775 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:56:09.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:56:09.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.124 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:56:09.778 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:56:09.778 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:56:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:56:09.778 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:56:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:56:09.778 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:56:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:56:09.778 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:56:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:56:09.778 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:56:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:56:09.778 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:56:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:56:09.778 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:56:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:56:09.778 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:56:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:56:09.778 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:56:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:56:09.778 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:56:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:56:09.778 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:56:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:56:09.778 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:56:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:56:09.778 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:56:09.778 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:56:09.778 139328 ERROR oslo_service.periodic_task >2018-06-24 17:56:39.650 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:56:39.651 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:56:39.657 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:56:39.658 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:56:39.691 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:56:39.691 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:56:39.691 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:56:39.691 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:56:39.691 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:56:39.691 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:56:39.691 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:56:39.693 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:56:39.693 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:56:39.694 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:56:49.694 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:56:49.727 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:56:49.727 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:56:49.727 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:56:49.727 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:56:49.727 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:56:49.727 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:56:49.727 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:56:49.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:56:49.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:56:49.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:57:09.730 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:57:09.763 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:57:09.763 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:57:09.763 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:57:09.763 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:57:09.763 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:57:09.763 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:57:09.763 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:57:09.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:57:09.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.107 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:57:09.766 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:57:09.766 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:57:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:57:09.766 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:57:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:57:09.766 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:57:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:57:09.766 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:57:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:57:09.766 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:57:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:57:09.766 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:57:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:57:09.766 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:57:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:57:09.766 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:57:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:57:09.766 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:57:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:57:09.766 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:57:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:57:09.766 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:57:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:57:09.766 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:57:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:57:09.766 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:57:09.766 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:57:09.766 139328 ERROR oslo_service.periodic_task >2018-06-24 17:57:39.647 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:57:39.648 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:57:39.655 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:57:39.656 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:57:39.689 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:57:39.689 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:57:39.689 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:57:39.689 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:57:39.689 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:57:39.689 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:57:39.689 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:57:39.691 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:57:39.692 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:57:39.692 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:57:49.693 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:57:49.724 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:57:49.724 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:57:49.724 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:57:49.724 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:57:49.724 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:57:49.724 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:57:49.724 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:57:49.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:57:49.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.07 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:57:49.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:58:09.726 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:58:09.759 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:58:09.759 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:58:09.759 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:58:09.759 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:58:09.759 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:58:09.759 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:58:09.759 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:58:09.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:58:09.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.105 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:58:09.761 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:58:09.761 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:58:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:58:09.761 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:58:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:58:09.761 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:58:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:58:09.761 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:58:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:58:09.761 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:58:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:58:09.761 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:58:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:58:09.761 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:58:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:58:09.761 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:58:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:58:09.761 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:58:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:58:09.761 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:58:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:58:09.761 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:58:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:58:09.761 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:58:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:58:09.761 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:58:09.761 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:58:09.761 139328 ERROR oslo_service.periodic_task >2018-06-24 17:58:39.643 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:58:39.643 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:58:39.650 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:58:39.651 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:58:39.686 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:58:39.686 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:58:39.686 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:58:39.686 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:58:39.686 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:58:39.686 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:58:39.686 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:58:39.688 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:58:39.689 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:58:39.689 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:58:49.689 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:58:49.724 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:58:49.724 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:58:49.724 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:58:49.724 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:58:49.724 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:58:49.724 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:58:49.724 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:58:49.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:58:49.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:58:49.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:59:09.728 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:59:09.765 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:59:09.765 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:59:09.765 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:59:09.765 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:59:09.765 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:59:09.765 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:59:09.765 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:59:09.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:59:09.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:59:09.769 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:59:09.769 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 17:59:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 17:59:09.769 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 17:59:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 17:59:09.769 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 17:59:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 17:59:09.769 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 17:59:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 17:59:09.769 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 17:59:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 17:59:09.769 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 17:59:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 17:59:09.769 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 17:59:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 17:59:09.769 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 17:59:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 17:59:09.769 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 17:59:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 17:59:09.769 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 17:59:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 17:59:09.769 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 17:59:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 17:59:09.769 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 17:59:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 17:59:09.769 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 17:59:09.769 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 17:59:09.769 139328 ERROR oslo_service.periodic_task >2018-06-24 17:59:39.640 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:59:39.641 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 17:59:39.648 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 17:59:39.649 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:59:39.683 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:59:39.683 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:59:39.683 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:59:39.683 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:59:39.683 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:59:39.683 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:59:39.683 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:59:39.686 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:59:39.687 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:59:39.687 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 17:59:49.688 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 17:59:49.731 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 17:59:49.731 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 17:59:49.731 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 17:59:49.731 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 17:59:49.731 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 17:59:49.731 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 17:59:49.731 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 17:59:49.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 17:59:49.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.086 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 17:59:49.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:00:09.736 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:00:09.775 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:00:09.775 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:00:09.775 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:00:09.775 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:00:09.775 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:00:09.775 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:00:09.775 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:00:09.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:00:09.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.13 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:00:09.779 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:00:09.779 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:00:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:00:09.779 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:00:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:00:09.779 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:00:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:00:09.779 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:00:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:00:09.779 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:00:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:00:09.779 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:00:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:00:09.779 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:00:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:00:09.779 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:00:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:00:09.779 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:00:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:00:09.779 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:00:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:00:09.779 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:00:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:00:09.779 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:00:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:00:09.779 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:00:09.779 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:00:09.779 139328 ERROR oslo_service.periodic_task >2018-06-24 18:00:39.640 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:00:39.641 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:00:39.651 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:00:39.651 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:00:39.687 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:00:39.687 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:00:39.687 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:00:39.687 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:00:39.687 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:00:39.687 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:00:39.687 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:00:39.689 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:00:39.690 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:00:39.690 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:00:49.690 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:00:49.724 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:00:49.724 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:00:49.724 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:00:49.724 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:00:49.724 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:00:49.724 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:00:49.724 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:00:49.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:00:49.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:00:49.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:01:09.727 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:01:09.763 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:01:09.763 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:01:09.763 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:01:09.763 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:01:09.763 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:01:09.763 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:01:09.763 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:01:09.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:01:09.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:01:09.766 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:01:09.766 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:01:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:01:09.766 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:01:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:01:09.766 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:01:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:01:09.766 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:01:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:01:09.766 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:01:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:01:09.766 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:01:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:01:09.766 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:01:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:01:09.766 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:01:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:01:09.766 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:01:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:01:09.766 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:01:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:01:09.766 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:01:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:01:09.766 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:01:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:01:09.766 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:01:09.766 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:01:09.766 139328 ERROR oslo_service.periodic_task >2018-06-24 18:01:39.638 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:01:39.639 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:01:39.650 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:01:39.650 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:01:39.684 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:01:39.684 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:01:39.684 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:01:39.684 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:01:39.684 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:01:39.684 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:01:39.684 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:01:39.686 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:01:39.687 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:01:39.687 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:01:49.687 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:01:49.720 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:01:49.720 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:01:49.720 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:01:49.720 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:01:49.720 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:01:49.720 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:01:49.720 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:01:49.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:01:49.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:01:49.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:02:09.723 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:02:09.756 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:02:09.756 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:02:09.756 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:02:09.756 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:02:09.756 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:02:09.756 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:02:09.756 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:02:09.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:02:09.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:02:09.759 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:02:09.759 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:02:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:02:09.759 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:02:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:02:09.759 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:02:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:02:09.759 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:02:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:02:09.759 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:02:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:02:09.759 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:02:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:02:09.759 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:02:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:02:09.759 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:02:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:02:09.759 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:02:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:02:09.759 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:02:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:02:09.759 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:02:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:02:09.759 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:02:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:02:09.759 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:02:09.759 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:02:09.759 139328 ERROR oslo_service.periodic_task >2018-06-24 18:02:39.640 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:02:39.642 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:02:39.648 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:02:39.649 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:02:39.682 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:02:39.682 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:02:39.682 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:02:39.682 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:02:39.682 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:02:39.682 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:02:39.682 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:02:39.684 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:02:39.684 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:02:39.685 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:02:49.685 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:02:49.718 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:02:49.718 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:02:49.718 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:02:49.718 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:02:49.718 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:02:49.718 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:02:49.718 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:02:49.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:02:49.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:02:49.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:03:09.722 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:03:09.756 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:03:09.756 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:03:09.756 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:03:09.756 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:03:09.756 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:03:09.756 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:03:09.756 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:03:09.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:03:09.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:03:09.758 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:03:09.758 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:03:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:03:09.758 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:03:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:03:09.758 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:03:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:03:09.758 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:03:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:03:09.758 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:03:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:03:09.758 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:03:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:03:09.758 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:03:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:03:09.758 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:03:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:03:09.758 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:03:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:03:09.758 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:03:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:03:09.758 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:03:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:03:09.758 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:03:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:03:09.758 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:03:09.758 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:03:09.758 139328 ERROR oslo_service.periodic_task >2018-06-24 18:03:39.639 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:03:39.641 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:03:39.654 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:03:39.655 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:03:39.687 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:03:39.687 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:03:39.687 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:03:39.687 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:03:39.687 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:03:39.687 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:03:39.687 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:03:39.689 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:03:39.690 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:03:39.690 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:03:49.690 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:03:49.724 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:03:49.724 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:03:49.724 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:03:49.724 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:03:49.724 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:03:49.724 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:03:49.724 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:03:49.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:03:49.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:03:49.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:04:09.727 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:04:09.760 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:04:09.760 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:04:09.760 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:04:09.760 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:04:09.760 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:04:09.760 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:04:09.760 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:04:09.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:04:09.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:04:09.763 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:04:09.763 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:04:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:04:09.763 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:04:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:04:09.763 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:04:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:04:09.763 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:04:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:04:09.763 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:04:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:04:09.763 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:04:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:04:09.763 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:04:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:04:09.763 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:04:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:04:09.763 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:04:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:04:09.763 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:04:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:04:09.763 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:04:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:04:09.763 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:04:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:04:09.763 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:04:09.763 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:04:09.763 139328 ERROR oslo_service.periodic_task >2018-06-24 18:04:39.635 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:04:39.636 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:04:39.649 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:04:39.649 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:04:39.699 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:04:39.699 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:04:39.699 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:04:39.699 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:04:39.699 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:04:39.699 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:04:39.699 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:04:39.702 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:04:39.703 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.053 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:04:39.703 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:04:49.704 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:04:49.740 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:04:49.740 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:04:49.740 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:04:49.740 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:04:49.740 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:04:49.740 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:04:49.740 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:04:49.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:04:49.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.092 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:04:49.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:05:09.743 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:05:09.778 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:05:09.778 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:05:09.778 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:05:09.778 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:05:09.778 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:05:09.778 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:05:09.778 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:05:09.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:05:09.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.13 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:05:09.780 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:05:09.780 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:05:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:05:09.780 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:05:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:05:09.780 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:05:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:05:09.780 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:05:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:05:09.780 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:05:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:05:09.780 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:05:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:05:09.780 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:05:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:05:09.780 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:05:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:05:09.780 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:05:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:05:09.780 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:05:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:05:09.780 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:05:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:05:09.780 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:05:09.780 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:05:09.780 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:05:09.780 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:05:09.780 139328 ERROR oslo_service.periodic_task >2018-06-24 18:05:39.632 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:05:39.633 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:05:39.641 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:05:39.642 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:05:39.675 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:05:39.675 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:05:39.675 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:05:39.675 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:05:39.675 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:05:39.675 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:05:39.675 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:05:39.678 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:05:39.678 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:05:39.679 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:05:49.679 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:05:49.711 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:05:49.711 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:05:49.711 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:05:49.711 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:05:49.711 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:05:49.711 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:05:49.711 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:05:49.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:05:49.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:05:49.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:06:09.714 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:06:09.748 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:06:09.748 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:06:09.748 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:06:09.748 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:06:09.748 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:06:09.748 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:06:09.748 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:06:09.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:06:09.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:06:09.751 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:06:09.751 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:06:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:06:09.751 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:06:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:06:09.751 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:06:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:06:09.751 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:06:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:06:09.751 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:06:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:06:09.751 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:06:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:06:09.751 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:06:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:06:09.751 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:06:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:06:09.751 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:06:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:06:09.751 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:06:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:06:09.751 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:06:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:06:09.751 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:06:09.751 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:06:09.751 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:06:09.751 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:06:09.751 139328 ERROR oslo_service.periodic_task >2018-06-24 18:06:39.632 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:06:39.633 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:06:39.640 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:06:39.640 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:06:39.672 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:06:39.672 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:06:39.672 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:06:39.672 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:06:39.672 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:06:39.672 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:06:39.672 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:06:39.674 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:06:39.674 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.033 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:06:39.674 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:06:49.675 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:06:49.711 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:06:49.711 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:06:49.711 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:06:49.711 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:06:49.711 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:06:49.711 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:06:49.711 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:06:49.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:06:49.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:06:49.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:07:09.715 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:07:09.749 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:07:09.749 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:07:09.749 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:07:09.749 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:07:09.749 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:07:09.749 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:07:09.749 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:07:09.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:07:09.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:07:09.752 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:07:09.752 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:07:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:07:09.752 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:07:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:07:09.752 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:07:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:07:09.752 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:07:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:07:09.752 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:07:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:07:09.752 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:07:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:07:09.752 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:07:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:07:09.752 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:07:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:07:09.752 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:07:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:07:09.752 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:07:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:07:09.752 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:07:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:07:09.752 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:07:09.752 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:07:09.752 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:07:09.752 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:07:09.752 139328 ERROR oslo_service.periodic_task >2018-06-24 18:07:39.633 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:07:39.635 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:07:39.643 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:07:39.644 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:07:39.678 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:07:39.678 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:07:39.678 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:07:39.678 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:07:39.678 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:07:39.678 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:07:39.678 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:07:39.680 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:07:39.680 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:07:39.681 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:07:49.681 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:07:49.715 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:07:49.715 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:07:49.715 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:07:49.715 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:07:49.715 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:07:49.715 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:07:49.715 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:07:49.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:07:49.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:07:49.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:08:09.719 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:08:09.751 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:08:09.751 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:08:09.751 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:08:09.751 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:08:09.751 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:08:09.751 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:08:09.751 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:08:09.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:08:09.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:08:09.753 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:08:09.753 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:08:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:08:09.753 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:08:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:08:09.753 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:08:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:08:09.753 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:08:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:08:09.753 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:08:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:08:09.753 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:08:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:08:09.753 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:08:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:08:09.753 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:08:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:08:09.753 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:08:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:08:09.753 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:08:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:08:09.753 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:08:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:08:09.753 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:08:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:08:09.753 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:08:09.753 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:08:09.753 139328 ERROR oslo_service.periodic_task >2018-06-24 18:08:39.635 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:08:39.635 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:08:39.645 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:08:39.646 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:08:39.681 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:08:39.681 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:08:39.681 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:08:39.681 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:08:39.681 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:08:39.681 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:08:39.681 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:08:39.683 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:08:39.683 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:08:39.684 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:08:49.684 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:08:49.719 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:08:49.719 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:08:49.719 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:08:49.719 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:08:49.719 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:08:49.719 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:08:49.719 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:08:49.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:08:49.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:08:49.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:09:09.723 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:09:09.754 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:09:09.754 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:09:09.754 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:09:09.754 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:09:09.754 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:09:09.754 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:09:09.754 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:09:09.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:09:09.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:09:09.757 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:09:09.757 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:09:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:09:09.757 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:09:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:09:09.757 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:09:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:09:09.757 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:09:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:09:09.757 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:09:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:09:09.757 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:09:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:09:09.757 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:09:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:09:09.757 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:09:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:09:09.757 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:09:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:09:09.757 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:09:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:09:09.757 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:09:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:09:09.757 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:09:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:09:09.757 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:09:09.757 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:09:09.757 139328 ERROR oslo_service.periodic_task >2018-06-24 18:09:39.638 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:09:39.640 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:09:39.647 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:09:39.647 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:09:39.681 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:09:39.681 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:09:39.681 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:09:39.681 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:09:39.681 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:09:39.681 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:09:39.681 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:09:39.683 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:09:39.683 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:09:39.684 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:09:49.684 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:09:49.717 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:09:49.717 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:09:49.717 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:09:49.717 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:09:49.717 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:09:49.717 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:09:49.717 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:09:49.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:09:49.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:09:49.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:10:09.721 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:10:09.753 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:10:09.753 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:10:09.753 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:10:09.753 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:10:09.753 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:10:09.753 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:10:09.753 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:10:09.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:10:09.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:10:09.756 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:10:09.756 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:10:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:10:09.756 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:10:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:10:09.756 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:10:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:10:09.756 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:10:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:10:09.756 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:10:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:10:09.756 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:10:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:10:09.756 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:10:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:10:09.756 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:10:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:10:09.756 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:10:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:10:09.756 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:10:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:10:09.756 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:10:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:10:09.756 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:10:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:10:09.756 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:10:09.756 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:10:09.756 139328 ERROR oslo_service.periodic_task >2018-06-24 18:10:39.637 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:10:39.638 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:10:39.651 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:10:39.652 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:10:39.684 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:10:39.684 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:10:39.684 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:10:39.684 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:10:39.684 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:10:39.684 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:10:39.684 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:10:39.685 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:10:39.686 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:10:39.686 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:10:49.687 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:10:49.720 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:10:49.720 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:10:49.720 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:10:49.720 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:10:49.720 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:10:49.720 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:10:49.720 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:10:49.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:10:49.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.07 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:10:49.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:11:09.723 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:11:09.757 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:11:09.757 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:11:09.757 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:11:09.757 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:11:09.757 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:11:09.757 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:11:09.757 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:11:09.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:11:09.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.107 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:11:09.759 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:11:09.759 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:11:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:11:09.759 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:11:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:11:09.759 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:11:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:11:09.759 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:11:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:11:09.759 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:11:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:11:09.759 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:11:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:11:09.759 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:11:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:11:09.759 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:11:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:11:09.759 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:11:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:11:09.759 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:11:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:11:09.759 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:11:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:11:09.759 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:11:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:11:09.759 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:11:09.759 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:11:09.759 139328 ERROR oslo_service.periodic_task >2018-06-24 18:11:39.641 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:11:39.642 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:11:39.654 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:11:39.655 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:11:39.691 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:11:39.691 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:11:39.691 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:11:39.691 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:11:39.691 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:11:39.691 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:11:39.691 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:11:39.694 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:11:39.694 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:11:39.694 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:11:49.695 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:11:49.731 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:11:49.731 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:11:49.731 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:11:49.731 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:11:49.731 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:11:49.731 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:11:49.731 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:11:49.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:11:49.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.08 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:11:49.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:12:09.736 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:12:09.772 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:12:09.772 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:12:09.772 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:12:09.772 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:12:09.772 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:12:09.772 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:12:09.772 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:12:09.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:12:09.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.12 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:12:09.775 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:12:09.775 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:12:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:12:09.775 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:12:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:12:09.775 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:12:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:12:09.775 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:12:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:12:09.775 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:12:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:12:09.775 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:12:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:12:09.775 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:12:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:12:09.775 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:12:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:12:09.775 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:12:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:12:09.775 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:12:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:12:09.775 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:12:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:12:09.775 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:12:09.775 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:12:09.775 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:12:09.775 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:12:09.775 139328 ERROR oslo_service.periodic_task >2018-06-24 18:12:39.636 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:12:39.637 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:12:39.645 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:12:39.645 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:12:39.680 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:12:39.680 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:12:39.680 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:12:39.680 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:12:39.680 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:12:39.680 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:12:39.680 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:12:39.682 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:12:39.682 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:12:39.683 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:12:49.683 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:12:49.719 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:12:49.719 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:12:49.719 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:12:49.719 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:12:49.719 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:12:49.719 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:12:49.719 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:12:49.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:12:49.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:12:49.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:13:09.723 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:13:09.755 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:13:09.755 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:13:09.755 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:13:09.755 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:13:09.755 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:13:09.755 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:13:09.755 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:13:09.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:13:09.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:13:09.758 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:13:09.758 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:13:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:13:09.758 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:13:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:13:09.758 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:13:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:13:09.758 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:13:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:13:09.758 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:13:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:13:09.758 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:13:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:13:09.758 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:13:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:13:09.758 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:13:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:13:09.758 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:13:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:13:09.758 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:13:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:13:09.758 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:13:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:13:09.758 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:13:09.758 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:13:09.758 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:13:09.758 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:13:09.758 139328 ERROR oslo_service.periodic_task >2018-06-24 18:13:39.639 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:13:39.641 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:13:39.647 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:13:39.648 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:13:39.681 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:13:39.681 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:13:39.681 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:13:39.681 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:13:39.681 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:13:39.681 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:13:39.681 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:13:39.684 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:13:39.684 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:13:39.685 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:13:49.685 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:13:49.731 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:13:49.731 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:13:49.731 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:13:49.731 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:13:49.731 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:13:49.731 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:13:49.731 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:13:49.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:13:49.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.085 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:13:49.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:14:09.734 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:14:09.766 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:14:09.766 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:14:09.766 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:14:09.766 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:14:09.766 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:14:09.766 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:14:09.766 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:14:09.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:14:09.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.121 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:14:09.769 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:14:09.769 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:14:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:14:09.769 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:14:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:14:09.769 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:14:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:14:09.769 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:14:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:14:09.769 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:14:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:14:09.769 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:14:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:14:09.769 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:14:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:14:09.769 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:14:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:14:09.769 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:14:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:14:09.769 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:14:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:14:09.769 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:14:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:14:09.769 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:14:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:14:09.769 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:14:09.769 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:14:09.769 139328 ERROR oslo_service.periodic_task >2018-06-24 18:14:39.640 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:14:39.641 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:14:39.649 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:14:39.650 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:14:39.696 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:14:39.696 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:14:39.696 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:14:39.696 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:14:39.696 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:14:39.696 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:14:39.696 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:14:39.698 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:14:39.698 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.048 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:14:39.699 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:14:49.699 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:14:49.733 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:14:49.733 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:14:49.733 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:14:49.733 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:14:49.733 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:14:49.733 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:14:49.733 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:14:49.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:14:49.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.085 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:14:49.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:15:09.736 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:15:09.774 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:15:09.774 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:15:09.774 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:15:09.774 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:15:09.774 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:15:09.774 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:15:09.774 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:15:09.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:15:09.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.125 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:15:09.777 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:15:09.777 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:15:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:15:09.777 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:15:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:15:09.777 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:15:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:15:09.777 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:15:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:15:09.777 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:15:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:15:09.777 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:15:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:15:09.777 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:15:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:15:09.777 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:15:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:15:09.777 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:15:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:15:09.777 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:15:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:15:09.777 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:15:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:15:09.777 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:15:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:15:09.777 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:15:09.777 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:15:09.777 139328 ERROR oslo_service.periodic_task >2018-06-24 18:15:39.638 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:15:39.639 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:15:39.646 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:15:39.647 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:15:39.685 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:15:39.685 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:15:39.685 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:15:39.685 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:15:39.685 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:15:39.685 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:15:39.685 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:15:39.688 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:15:39.688 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.041 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:15:39.688 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:15:49.689 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:15:49.723 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:15:49.723 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:15:49.723 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:15:49.723 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:15:49.723 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:15:49.723 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:15:49.723 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:15:49.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:15:49.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:15:49.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:16:09.727 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:16:09.760 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:16:09.760 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:16:09.760 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:16:09.760 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:16:09.760 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:16:09.760 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:16:09.760 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:16:09.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:16:09.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:16:09.763 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:16:09.763 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:16:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:16:09.763 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:16:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:16:09.763 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:16:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:16:09.763 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:16:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:16:09.763 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:16:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:16:09.763 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:16:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:16:09.763 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:16:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:16:09.763 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:16:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:16:09.763 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:16:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:16:09.763 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:16:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:16:09.763 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:16:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:16:09.763 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:16:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:16:09.763 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:16:09.763 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:16:09.763 139328 ERROR oslo_service.periodic_task >2018-06-24 18:16:39.634 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:16:39.636 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:16:39.643 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:16:39.644 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:16:39.678 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:16:39.678 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:16:39.678 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:16:39.678 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:16:39.678 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:16:39.678 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:16:39.678 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:16:39.681 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:16:39.681 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:16:39.681 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:16:49.682 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:16:49.715 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:16:49.715 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:16:49.715 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:16:49.715 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:16:49.715 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:16:49.715 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:16:49.715 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:16:49.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:16:49.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:16:49.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:17:09.718 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:17:09.751 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:17:09.751 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:17:09.751 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:17:09.751 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:17:09.751 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:17:09.751 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:17:09.751 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:17:09.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:17:09.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:17:09.754 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:17:09.754 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:17:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:17:09.754 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:17:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:17:09.754 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:17:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:17:09.754 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:17:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:17:09.754 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:17:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:17:09.754 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:17:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:17:09.754 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:17:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:17:09.754 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:17:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:17:09.754 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:17:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:17:09.754 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:17:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:17:09.754 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:17:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:17:09.754 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:17:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:17:09.754 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:17:09.754 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:17:09.754 139328 ERROR oslo_service.periodic_task >2018-06-24 18:17:39.636 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:17:39.637 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:17:39.643 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:17:39.644 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:17:39.676 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:17:39.676 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:17:39.676 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:17:39.676 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:17:39.676 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:17:39.676 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:17:39.676 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:17:39.678 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:17:39.678 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:17:39.679 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:17:49.679 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:17:49.713 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:17:49.713 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:17:49.713 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:17:49.713 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:17:49.713 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:17:49.713 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:17:49.713 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:17:49.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:17:49.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:17:49.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:18:09.716 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:18:09.763 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:18:09.763 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:18:09.763 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:18:09.763 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:18:09.763 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:18:09.763 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:18:09.763 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:18:09.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:18:09.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.122 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:18:09.766 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:18:09.766 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:18:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:18:09.766 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:18:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:18:09.766 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:18:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:18:09.766 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:18:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:18:09.766 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:18:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:18:09.766 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:18:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:18:09.766 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:18:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:18:09.766 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:18:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:18:09.766 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:18:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:18:09.766 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:18:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:18:09.766 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:18:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:18:09.766 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:18:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:18:09.766 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:18:09.766 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:18:09.766 139328 ERROR oslo_service.periodic_task >2018-06-24 18:18:39.637 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:18:39.639 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:18:39.646 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:18:39.647 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:18:39.679 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:18:39.679 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:18:39.679 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:18:39.679 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:18:39.679 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:18:39.679 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:18:39.679 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:18:39.681 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:18:39.682 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:18:39.682 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:18:49.683 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:18:49.717 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:18:49.717 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:18:49.717 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:18:49.717 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:18:49.717 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:18:49.717 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:18:49.717 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:18:49.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:18:49.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:18:49.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:19:09.720 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:19:09.753 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:19:09.753 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:19:09.753 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:19:09.753 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:19:09.753 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:19:09.753 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:19:09.753 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:19:09.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:19:09.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:19:09.756 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:19:09.756 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:19:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:19:09.756 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:19:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:19:09.756 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:19:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:19:09.756 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:19:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:19:09.756 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:19:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:19:09.756 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:19:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:19:09.756 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:19:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:19:09.756 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:19:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:19:09.756 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:19:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:19:09.756 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:19:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:19:09.756 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:19:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:19:09.756 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:19:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:19:09.756 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:19:09.756 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:19:09.756 139328 ERROR oslo_service.periodic_task >2018-06-24 18:19:39.637 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:19:39.638 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:19:39.645 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:19:39.645 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:19:39.678 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:19:39.678 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:19:39.678 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:19:39.678 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:19:39.678 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:19:39.678 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:19:39.678 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:19:39.680 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:19:39.680 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:19:39.681 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:19:49.682 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:19:49.714 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:19:49.714 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:19:49.714 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:19:49.714 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:19:49.714 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:19:49.714 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:19:49.714 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:19:49.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:19:49.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:19:49.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:20:09.718 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:20:09.751 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:20:09.751 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:20:09.751 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:20:09.751 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:20:09.751 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:20:09.751 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:20:09.751 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:20:09.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:20:09.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.107 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:20:09.753 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:20:09.753 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:20:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:20:09.753 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:20:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:20:09.753 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:20:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:20:09.753 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:20:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:20:09.753 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:20:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:20:09.753 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:20:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:20:09.753 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:20:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:20:09.753 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:20:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:20:09.753 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:20:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:20:09.753 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:20:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:20:09.753 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:20:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:20:09.753 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:20:09.753 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:20:09.753 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:20:09.753 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:20:09.753 139328 ERROR oslo_service.periodic_task >2018-06-24 18:20:39.635 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:20:39.636 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:21:39.634 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:21:39.635 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:21:39.642 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:21:39.643 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:21:39.676 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:21:39.676 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:21:39.676 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:21:39.676 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:21:39.676 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:21:39.676 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:21:39.676 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:21:39.678 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:21:39.678 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:21:39.679 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:21:49.679 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:21:49.714 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:21:49.714 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:21:49.714 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:21:49.714 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:21:49.714 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:21:49.714 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:21:49.714 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:21:49.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:21:49.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:21:49.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:22:09.717 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:22:09.752 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:22:09.752 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:22:09.752 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:22:09.752 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:22:09.752 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:22:09.752 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:22:09.752 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:22:09.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:22:09.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:22:09.754 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:22:09.754 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:22:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:22:09.754 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:22:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:22:09.754 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:22:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:22:09.754 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:22:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:22:09.754 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:22:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:22:09.754 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:22:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:22:09.754 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:22:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:22:09.754 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:22:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:22:09.754 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:22:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:22:09.754 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:22:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:22:09.754 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:22:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:22:09.754 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:22:09.754 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:22:09.754 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:22:09.754 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:22:09.754 139328 ERROR oslo_service.periodic_task >2018-06-24 18:22:39.636 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:22:39.637 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:22:39.646 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:22:39.646 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:22:39.689 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:22:39.689 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:22:39.689 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:22:39.689 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:22:39.689 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:22:39.689 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:22:39.689 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:22:39.692 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:22:39.692 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.045 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:22:39.693 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:22:49.693 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:22:49.725 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:22:49.725 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:22:49.725 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:22:49.725 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:22:49.725 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:22:49.725 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:22:49.725 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:22:49.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:22:49.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.081 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:22:49.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:23:09.729 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:23:09.762 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:23:09.762 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:23:09.762 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:23:09.762 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:23:09.762 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:23:09.762 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:23:09.762 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:23:09.764 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:23:09.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.118 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:23:09.765 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:23:09.765 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:23:09.765 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:23:09.765 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:23:09.765 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:23:09.765 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:23:09.765 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:23:09.765 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:23:09.765 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:23:09.765 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:23:09.765 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:23:09.765 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:23:09.765 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:23:09.765 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:23:09.765 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:23:09.765 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:23:09.765 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:23:09.765 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:23:09.765 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:23:09.765 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:23:09.765 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:23:09.765 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:23:09.765 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:23:09.765 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:23:09.765 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:23:09.765 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:23:09.765 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:23:09.765 139328 ERROR oslo_service.periodic_task >2018-06-24 18:23:39.636 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:23:39.637 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:23:39.646 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:23:39.647 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:23:39.681 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:23:39.681 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:23:39.681 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:23:39.681 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:23:39.681 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:23:39.681 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:23:39.681 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:23:39.683 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:23:39.683 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:23:39.684 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:23:49.684 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:23:49.714 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:23:49.714 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:23:49.714 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:23:49.714 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:23:49.714 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:23:49.714 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:23:49.714 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:23:49.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:23:49.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.068 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:23:49.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:24:09.717 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:24:09.758 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:24:09.758 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:24:09.758 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:24:09.758 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:24:09.758 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:24:09.758 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:24:09.758 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:24:09.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:24:09.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:24:09.761 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:24:09.761 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:24:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:24:09.761 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:24:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:24:09.761 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:24:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:24:09.761 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:24:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:24:09.761 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:24:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:24:09.761 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:24:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:24:09.761 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:24:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:24:09.761 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:24:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:24:09.761 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:24:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:24:09.761 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:24:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:24:09.761 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:24:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:24:09.761 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:24:09.761 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:24:09.761 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:24:09.761 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:24:09.761 139328 ERROR oslo_service.periodic_task >2018-06-24 18:24:39.632 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:24:39.633 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:24:39.642 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:24:39.643 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:24:39.677 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:24:39.677 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:24:39.677 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:24:39.677 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:24:39.677 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:24:39.677 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:24:39.677 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:24:39.680 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:24:39.680 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:24:39.681 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:24:49.681 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:24:49.715 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:24:49.715 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:24:49.715 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:24:49.715 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:24:49.715 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:24:49.715 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:24:49.715 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:24:49.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:24:49.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:24:49.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:25:09.718 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:25:09.752 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:25:09.752 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:25:09.752 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:25:09.752 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:25:09.752 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:25:09.752 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:25:09.752 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:25:09.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:25:09.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:25:09.755 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:25:09.755 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:25:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:25:09.755 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:25:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:25:09.755 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:25:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:25:09.755 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:25:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:25:09.755 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:25:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:25:09.755 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:25:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:25:09.755 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:25:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:25:09.755 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:25:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:25:09.755 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:25:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:25:09.755 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:25:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:25:09.755 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:25:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:25:09.755 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:25:09.755 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:25:09.755 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:25:09.755 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:25:09.755 139328 ERROR oslo_service.periodic_task >2018-06-24 18:25:39.637 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:25:39.638 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:25:39.646 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:25:39.646 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:25:39.683 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:25:39.683 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:25:39.683 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:25:39.683 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:25:39.683 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:25:39.683 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:25:39.683 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:25:39.686 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:25:39.686 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:25:39.686 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:25:49.687 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:25:49.720 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:25:49.720 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:25:49.720 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:25:49.720 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:25:49.720 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:25:49.720 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:25:49.720 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:25:49.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:25:49.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:25:49.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:26:09.723 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:26:09.756 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:26:09.756 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:26:09.756 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:26:09.756 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:26:09.756 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:26:09.756 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:26:09.756 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:26:09.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:26:09.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:26:09.759 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:26:09.759 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:26:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:26:09.759 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:26:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:26:09.759 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:26:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:26:09.759 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:26:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:26:09.759 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:26:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:26:09.759 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:26:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:26:09.759 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:26:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:26:09.759 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:26:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:26:09.759 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:26:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:26:09.759 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:26:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:26:09.759 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:26:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:26:09.759 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:26:09.759 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:26:09.759 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:26:09.759 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:26:09.759 139328 ERROR oslo_service.periodic_task >2018-06-24 18:26:39.640 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:26:39.642 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:26:39.650 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:26:39.651 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:26:39.689 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:26:39.689 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:26:39.689 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:26:39.689 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:26:39.689 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:26:39.689 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:26:39.689 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:26:39.692 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:26:39.692 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:26:39.692 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:26:49.693 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:26:49.728 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:26:49.728 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:26:49.728 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:26:49.728 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:26:49.728 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:26:49.728 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:26:49.728 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:26:49.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:26:49.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:26:49.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:27:09.731 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:27:09.768 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:27:09.768 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:27:09.768 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:27:09.768 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:27:09.768 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:27:09.768 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:27:09.768 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:27:09.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:27:09.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.119 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:27:09.772 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:27:09.772 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:27:09.772 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:27:09.772 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:27:09.772 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:27:09.772 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:27:09.772 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:27:09.772 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:27:09.772 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:27:09.772 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:27:09.772 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:27:09.772 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:27:09.772 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:27:09.772 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:27:09.772 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:27:09.772 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:27:09.772 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:27:09.772 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:27:09.772 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:27:09.772 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:27:09.772 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:27:09.772 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:27:09.772 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:27:09.772 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:27:09.772 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:27:09.772 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:27:09.772 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:27:09.772 139328 ERROR oslo_service.periodic_task >2018-06-24 18:27:39.644 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:27:39.645 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:27:39.654 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:27:39.655 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:27:39.693 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:27:39.693 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:27:39.693 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:27:39.693 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:27:39.693 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:27:39.693 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:27:39.693 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:27:39.696 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:27:39.696 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:27:39.696 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:27:49.697 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:27:49.729 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:27:49.729 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:27:49.729 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:27:49.729 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:27:49.729 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:27:49.729 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:27:49.729 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:27:49.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:27:49.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:27:49.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:28:09.733 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:28:09.768 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:28:09.768 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:28:09.768 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:28:09.768 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:28:09.768 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:28:09.768 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:28:09.768 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:28:09.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:28:09.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:28:09.770 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:28:09.770 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:28:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:28:09.770 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:28:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:28:09.770 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:28:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:28:09.770 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:28:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:28:09.770 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:28:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:28:09.770 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:28:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:28:09.770 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:28:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:28:09.770 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:28:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:28:09.770 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:28:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:28:09.770 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:28:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:28:09.770 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:28:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:28:09.770 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:28:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:28:09.770 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:28:09.770 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:28:09.770 139328 ERROR oslo_service.periodic_task >2018-06-24 18:28:39.641 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:28:39.642 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:28:39.650 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:28:39.651 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:28:39.684 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:28:39.684 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:28:39.684 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:28:39.684 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:28:39.684 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:28:39.684 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:28:39.684 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:28:39.686 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:28:39.686 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:28:39.686 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:28:49.687 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:28:49.724 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:28:49.724 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:28:49.724 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:28:49.724 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:28:49.724 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:28:49.724 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:28:49.724 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:28:49.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:28:49.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:28:49.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:29:09.727 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:29:09.759 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:29:09.759 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:29:09.759 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:29:09.759 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:29:09.759 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:29:09.759 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:29:09.759 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:29:09.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:29:09.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:29:09.762 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:29:09.762 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:29:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:29:09.762 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:29:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:29:09.762 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:29:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:29:09.762 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:29:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:29:09.762 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:29:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:29:09.762 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:29:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:29:09.762 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:29:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:29:09.762 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:29:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:29:09.762 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:29:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:29:09.762 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:29:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:29:09.762 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:29:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:29:09.762 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:29:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:29:09.762 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:29:09.762 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:29:09.762 139328 ERROR oslo_service.periodic_task >2018-06-24 18:29:39.643 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:29:39.644 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:29:39.651 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:29:39.652 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:29:39.688 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:29:39.688 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:29:39.688 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:29:39.688 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:29:39.688 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:29:39.688 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:29:39.688 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:29:39.691 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:29:39.691 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:29:39.691 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:29:49.692 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:29:49.730 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:29:49.730 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:29:49.730 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:29:49.730 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:29:49.730 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:29:49.730 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:29:49.730 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:29:49.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:29:49.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.08 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:29:49.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:30:09.733 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:30:09.767 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:30:09.767 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:30:09.767 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:30:09.767 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:30:09.767 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:30:09.767 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:30:09.767 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:30:09.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:30:09.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:30:09.770 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:30:09.770 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:30:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:30:09.770 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:30:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:30:09.770 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:30:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:30:09.770 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:30:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:30:09.770 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:30:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:30:09.770 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:30:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:30:09.770 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:30:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:30:09.770 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:30:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:30:09.770 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:30:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:30:09.770 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:30:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:30:09.770 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:30:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:30:09.770 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:30:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:30:09.770 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:30:09.770 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:30:09.770 139328 ERROR oslo_service.periodic_task >2018-06-24 18:30:39.641 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:30:39.642 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:30:39.648 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:30:39.649 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:30:39.683 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:30:39.683 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:30:39.683 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:30:39.683 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:30:39.683 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:30:39.683 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:30:39.683 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:30:39.685 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:30:39.685 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:30:39.686 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:30:49.686 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:30:49.720 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:30:49.720 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:30:49.720 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:30:49.720 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:30:49.720 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:30:49.720 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:30:49.720 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:30:49.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:30:49.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:30:49.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:31:09.723 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:31:09.754 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:31:09.754 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:31:09.754 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:31:09.754 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:31:09.754 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:31:09.754 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:31:09.754 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:31:09.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:31:09.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.107 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:31:09.756 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:31:09.756 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:31:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:31:09.756 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:31:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:31:09.756 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:31:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:31:09.756 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:31:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:31:09.756 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:31:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:31:09.756 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:31:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:31:09.756 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:31:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:31:09.756 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:31:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:31:09.756 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:31:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:31:09.756 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:31:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:31:09.756 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:31:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:31:09.756 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:31:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:31:09.756 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:31:09.756 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:31:09.756 139328 ERROR oslo_service.periodic_task >2018-06-24 18:31:39.637 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:31:39.639 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:31:39.646 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:31:39.646 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:31:39.681 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:31:39.681 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:31:39.681 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:31:39.681 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:31:39.681 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:31:39.681 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:31:39.681 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:31:39.684 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:31:39.684 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:31:39.684 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:31:49.685 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:31:49.718 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:31:49.718 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:31:49.718 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:31:49.718 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:31:49.718 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:31:49.718 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:31:49.718 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:31:49.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:31:49.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:31:49.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:32:09.721 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:32:09.754 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:32:09.754 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:32:09.754 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:32:09.754 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:32:09.754 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:32:09.754 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:32:09.754 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:32:09.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:32:09.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:32:09.757 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:32:09.757 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:32:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:32:09.757 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:32:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:32:09.757 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:32:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:32:09.757 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:32:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:32:09.757 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:32:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:32:09.757 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:32:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:32:09.757 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:32:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:32:09.757 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:32:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:32:09.757 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:32:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:32:09.757 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:32:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:32:09.757 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:32:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:32:09.757 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:32:09.757 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:32:09.757 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:32:09.757 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:32:09.757 139328 ERROR oslo_service.periodic_task >2018-06-24 18:32:39.638 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:32:39.639 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:32:39.647 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:32:39.648 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:32:39.683 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:32:39.683 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:32:39.683 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:32:39.683 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:32:39.683 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:32:39.683 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:32:39.683 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:32:39.685 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:32:39.686 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:32:39.686 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:32:49.687 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:32:49.722 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:32:49.722 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:32:49.722 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:32:49.722 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:32:49.722 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:32:49.722 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:32:49.722 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:32:49.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:32:49.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:32:49.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:33:09.725 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:33:09.759 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:33:09.759 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:33:09.759 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:33:09.759 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:33:09.759 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:33:09.759 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:33:09.759 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:33:09.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:33:09.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:33:09.763 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:33:09.763 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:33:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:33:09.763 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:33:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:33:09.763 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:33:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:33:09.763 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:33:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:33:09.763 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:33:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:33:09.763 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:33:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:33:09.763 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:33:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:33:09.763 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:33:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:33:09.763 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:33:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:33:09.763 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:33:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:33:09.763 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:33:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:33:09.763 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:33:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:33:09.763 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:33:09.763 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:33:09.763 139328 ERROR oslo_service.periodic_task >2018-06-24 18:33:39.635 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:33:39.635 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:33:39.649 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:33:39.649 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:33:39.681 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:33:39.681 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:33:39.681 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:33:39.681 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:33:39.681 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:33:39.681 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:33:39.681 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:33:39.682 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:33:39.683 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.033 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:33:39.683 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:33:49.684 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:33:49.717 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:33:49.717 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:33:49.717 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:33:49.717 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:33:49.717 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:33:49.717 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:33:49.717 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:33:49.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:33:49.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.069 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:33:49.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:34:09.720 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:34:09.752 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:34:09.752 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:34:09.752 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:34:09.752 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:34:09.752 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:34:09.752 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:34:09.752 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:34:09.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:34:09.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.105 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:34:09.756 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:34:09.756 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:34:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:34:09.756 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:34:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:34:09.756 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:34:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:34:09.756 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:34:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:34:09.756 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:34:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:34:09.756 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:34:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:34:09.756 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:34:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:34:09.756 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:34:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:34:09.756 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:34:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:34:09.756 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:34:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:34:09.756 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:34:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:34:09.756 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:34:09.756 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:34:09.756 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:34:09.756 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:34:09.756 139328 ERROR oslo_service.periodic_task >2018-06-24 18:34:39.637 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:34:39.638 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:34:39.651 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:34:39.652 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:34:39.684 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:34:39.684 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:34:39.684 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:34:39.684 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:34:39.684 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:34:39.684 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:34:39.684 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:34:39.686 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:34:39.686 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:34:39.687 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:34:49.688 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:34:49.719 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:34:49.719 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:34:49.719 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:34:49.719 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:34:49.719 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:34:49.719 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:34:49.719 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:34:49.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:34:49.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:34:49.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:35:09.723 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:35:09.757 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:35:09.757 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:35:09.757 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:35:09.757 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:35:09.757 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:35:09.757 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:35:09.757 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:35:09.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:35:09.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:35:09.760 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:35:09.760 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:35:09.760 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:35:09.760 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:35:09.760 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:35:09.760 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:35:09.760 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:35:09.760 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:35:09.760 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:35:09.760 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:35:09.760 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:35:09.760 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:35:09.760 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:35:09.760 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:35:09.760 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:35:09.760 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:35:09.760 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:35:09.760 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:35:09.760 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:35:09.760 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:35:09.760 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:35:09.760 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:35:09.760 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:35:09.760 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:35:09.760 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:35:09.760 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:35:09.760 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:35:09.760 139328 ERROR oslo_service.periodic_task >2018-06-24 18:35:39.642 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:35:39.643 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:35:39.650 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:35:39.651 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:35:39.685 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:35:39.685 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:35:39.685 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:35:39.685 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:35:39.685 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:35:39.685 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:35:39.685 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:35:39.687 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:35:39.687 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:35:39.688 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:35:49.688 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:35:49.729 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:35:49.729 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:35:49.729 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:35:49.729 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:35:49.729 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:35:49.729 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:35:49.729 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:35:49.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:35:49.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.081 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:35:49.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:36:09.733 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:36:09.766 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:36:09.766 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:36:09.766 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:36:09.766 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:36:09.766 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:36:09.766 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:36:09.766 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:36:09.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:36:09.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.118 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:36:09.770 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:36:09.770 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:36:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:36:09.770 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:36:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:36:09.770 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:36:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:36:09.770 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:36:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:36:09.770 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:36:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:36:09.770 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:36:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:36:09.770 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:36:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:36:09.770 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:36:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:36:09.770 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:36:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:36:09.770 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:36:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:36:09.770 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:36:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:36:09.770 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:36:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:36:09.770 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:36:09.770 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:36:09.770 139328 ERROR oslo_service.periodic_task >2018-06-24 18:36:39.641 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:36:39.643 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:36:39.650 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:36:39.651 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:36:39.686 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:36:39.686 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:36:39.686 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:36:39.686 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:36:39.686 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:36:39.686 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:36:39.686 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:36:39.688 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:36:39.688 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:36:39.689 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:36:49.689 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:36:49.726 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:36:49.726 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:36:49.726 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:36:49.726 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:36:49.726 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:36:49.726 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:36:49.726 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:36:49.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:36:49.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:36:49.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:37:09.730 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:37:09.766 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:37:09.766 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:37:09.766 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:37:09.766 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:37:09.766 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:37:09.766 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:37:09.766 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:37:09.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:37:09.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.118 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:37:09.769 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:37:09.769 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:37:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:37:09.769 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:37:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:37:09.769 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:37:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:37:09.769 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:37:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:37:09.769 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:37:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:37:09.769 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:37:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:37:09.769 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:37:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:37:09.769 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:37:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:37:09.769 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:37:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:37:09.769 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:37:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:37:09.769 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:37:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:37:09.769 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:37:09.769 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:37:09.769 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:37:09.769 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:37:09.769 139328 ERROR oslo_service.periodic_task >2018-06-24 18:37:39.641 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:37:39.642 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:37:39.651 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:37:39.652 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:37:39.685 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:37:39.685 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:37:39.685 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:37:39.685 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:37:39.685 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:37:39.685 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:37:39.685 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:37:39.687 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:37:39.688 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:37:39.688 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:37:49.688 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:37:49.722 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:37:49.722 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:37:49.722 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:37:49.722 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:37:49.722 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:37:49.722 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:37:49.722 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:37:49.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:37:49.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:37:49.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:38:09.726 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:38:09.760 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:38:09.760 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:38:09.760 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:38:09.760 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:38:09.760 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:38:09.760 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:38:09.760 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:38:09.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:38:09.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:38:09.763 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:38:09.763 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:38:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:38:09.763 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:38:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:38:09.763 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:38:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:38:09.763 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:38:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:38:09.763 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:38:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:38:09.763 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:38:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:38:09.763 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:38:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:38:09.763 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:38:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:38:09.763 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:38:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:38:09.763 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:38:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:38:09.763 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:38:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:38:09.763 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:38:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:38:09.763 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:38:09.763 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:38:09.763 139328 ERROR oslo_service.periodic_task >2018-06-24 18:38:39.645 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:38:39.645 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:38:39.657 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:38:39.657 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:38:39.692 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:38:39.692 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:38:39.692 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:38:39.692 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:38:39.692 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:38:39.692 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:38:39.692 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:38:39.694 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:38:39.694 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:38:39.695 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:38:49.695 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:38:49.730 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:38:49.730 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:38:49.730 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:38:49.730 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:38:49.730 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:38:49.730 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:38:49.730 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:38:49.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:38:49.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:38:49.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:39:09.734 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:39:09.767 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:39:09.767 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:39:09.767 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:39:09.767 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:39:09.767 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:39:09.767 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:39:09.767 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:39:09.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:39:09.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:39:09.770 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:39:09.770 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:39:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:39:09.770 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:39:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:39:09.770 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:39:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:39:09.770 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:39:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:39:09.770 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:39:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:39:09.770 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:39:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:39:09.770 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:39:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:39:09.770 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:39:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:39:09.770 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:39:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:39:09.770 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:39:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:39:09.770 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:39:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:39:09.770 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:39:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:39:09.770 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:39:09.770 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:39:09.770 139328 ERROR oslo_service.periodic_task >2018-06-24 18:39:39.641 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:39:39.642 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:39:39.652 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:39:39.653 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:39:39.688 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:39:39.688 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:39:39.688 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:39:39.688 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:39:39.688 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:39:39.688 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:39:39.688 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:39:39.690 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:39:39.690 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:39:39.691 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:39:49.691 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:39:49.725 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:39:49.725 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:39:49.725 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:39:49.725 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:39:49.725 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:39:49.725 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:39:49.725 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:39:49.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:39:49.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:39:49.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:40:09.728 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:40:09.760 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:40:09.760 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:40:09.760 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:40:09.760 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:40:09.760 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:40:09.760 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:40:09.760 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:40:09.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:40:09.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:40:09.764 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:40:09.764 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:40:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:40:09.764 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:40:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:40:09.764 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:40:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:40:09.764 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:40:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:40:09.764 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:40:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:40:09.764 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:40:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:40:09.764 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:40:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:40:09.764 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:40:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:40:09.764 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:40:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:40:09.764 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:40:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:40:09.764 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:40:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:40:09.764 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:40:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:40:09.764 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:40:09.764 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:40:09.764 139328 ERROR oslo_service.periodic_task >2018-06-24 18:40:39.646 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:40:39.647 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:40:39.654 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:40:39.654 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:40:39.686 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:40:39.686 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:40:39.686 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:40:39.686 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:40:39.686 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:40:39.686 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:40:39.686 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:40:39.689 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:40:39.689 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:40:39.689 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:40:49.690 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:40:49.724 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:40:49.724 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:40:49.724 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:40:49.724 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:40:49.724 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:40:49.724 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:40:49.724 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:40:49.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:40:49.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:40:49.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:41:09.727 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:41:09.760 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:41:09.760 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:41:09.760 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:41:09.760 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:41:09.760 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:41:09.760 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:41:09.760 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:41:09.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:41:09.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.107 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:41:09.762 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:41:09.762 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:41:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:41:09.762 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:41:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:41:09.762 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:41:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:41:09.762 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:41:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:41:09.762 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:41:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:41:09.762 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:41:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:41:09.762 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:41:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:41:09.762 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:41:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:41:09.762 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:41:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:41:09.762 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:41:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:41:09.762 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:41:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:41:09.762 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:41:09.762 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:41:09.762 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:41:09.762 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:41:09.762 139328 ERROR oslo_service.periodic_task >2018-06-24 18:41:39.644 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:41:39.645 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:41:39.652 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:41:39.653 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:41:39.689 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:41:39.689 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:41:39.689 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:41:39.689 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:41:39.689 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:41:39.689 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:41:39.689 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:41:39.691 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:41:39.691 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:41:39.691 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:41:49.692 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:41:49.723 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:41:49.723 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:41:49.723 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:41:49.723 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:41:49.723 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:41:49.723 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:41:49.723 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:41:49.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:41:49.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:41:49.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:42:09.727 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:42:09.760 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:42:09.760 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:42:09.760 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:42:09.760 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:42:09.760 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:42:09.760 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:42:09.760 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:42:09.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:42:09.764 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:42:09.764 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:42:09.764 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:42:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:42:09.764 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:42:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:42:09.764 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:42:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:42:09.764 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:42:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:42:09.764 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:42:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:42:09.764 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:42:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:42:09.764 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:42:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:42:09.764 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:42:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:42:09.764 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:42:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:42:09.764 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:42:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:42:09.764 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:42:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:42:09.764 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:42:09.764 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:42:09.764 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:42:09.764 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:42:09.764 139328 ERROR oslo_service.periodic_task >2018-06-24 18:42:39.646 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:42:39.647 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:42:39.655 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:42:39.655 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:42:39.689 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:42:39.689 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:42:39.689 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:42:39.689 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:42:39.689 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:42:39.689 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:42:39.689 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:42:39.691 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:42:39.691 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:42:39.692 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:42:49.692 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:42:49.724 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:42:49.724 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:42:49.724 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:42:49.724 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:42:49.724 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:42:49.724 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:42:49.724 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:42:49.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:42:49.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.07 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:42:49.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:43:09.727 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:43:09.763 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:43:09.763 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:43:09.763 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:43:09.763 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:43:09.763 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:43:09.763 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:43:09.763 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:43:09.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:43:09.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:43:09.766 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:43:09.766 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:43:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:43:09.766 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:43:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:43:09.766 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:43:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:43:09.766 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:43:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:43:09.766 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:43:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:43:09.766 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:43:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:43:09.766 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:43:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:43:09.766 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:43:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:43:09.766 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:43:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:43:09.766 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:43:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:43:09.766 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:43:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:43:09.766 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:43:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:43:09.766 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:43:09.766 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:43:09.766 139328 ERROR oslo_service.periodic_task >2018-06-24 18:43:39.647 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:43:39.648 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:43:39.655 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:43:39.656 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:43:39.691 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:43:39.691 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:43:39.691 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:43:39.691 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:43:39.691 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:43:39.691 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:43:39.691 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:43:39.693 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:43:39.694 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:43:39.694 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:43:49.695 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:43:49.732 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:43:49.732 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:43:49.732 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:43:49.732 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:43:49.732 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:43:49.732 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:43:49.732 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:43:49.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:43:49.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:43:49.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:44:09.737 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:44:09.768 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:44:09.768 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:44:09.768 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:44:09.768 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:44:09.768 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:44:09.768 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:44:09.768 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:44:09.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:44:09.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:44:09.770 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:44:09.770 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:44:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:44:09.770 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:44:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:44:09.770 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:44:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:44:09.770 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:44:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:44:09.770 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:44:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:44:09.770 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:44:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:44:09.770 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:44:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:44:09.770 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:44:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:44:09.770 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:44:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:44:09.770 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:44:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:44:09.770 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:44:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:44:09.770 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:44:09.770 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:44:09.770 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:44:09.770 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:44:09.770 139328 ERROR oslo_service.periodic_task >2018-06-24 18:44:39.652 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:44:39.653 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:44:39.660 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:44:39.660 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:44:39.695 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:44:39.695 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:44:39.695 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:44:39.695 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:44:39.695 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:44:39.695 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:44:39.695 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:44:39.697 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:44:39.698 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:44:39.698 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:44:49.699 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:44:49.736 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:44:49.736 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:44:49.736 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:44:49.736 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:44:49.736 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:44:49.736 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:44:49.736 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:44:49.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:44:49.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:44:49.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:45:09.740 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:45:09.773 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:45:09.773 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:45:09.773 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:45:09.773 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:45:09.773 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:45:09.773 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:45:09.773 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:45:09.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:45:09.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:45:09.776 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:45:09.776 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:45:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:45:09.776 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:45:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:45:09.776 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:45:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:45:09.776 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:45:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:45:09.776 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:45:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:45:09.776 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:45:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:45:09.776 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:45:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:45:09.776 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:45:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:45:09.776 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:45:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:45:09.776 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:45:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:45:09.776 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:45:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:45:09.776 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:45:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:45:09.776 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:45:09.776 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:45:09.776 139328 ERROR oslo_service.periodic_task >2018-06-24 18:45:39.647 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:45:39.648 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:45:39.656 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:45:39.656 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:45:39.692 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:45:39.692 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:45:39.692 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:45:39.692 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:45:39.692 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:45:39.692 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:45:39.692 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:45:39.694 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:45:39.695 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:45:39.695 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:45:49.696 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:45:49.728 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:45:49.728 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:45:49.728 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:45:49.728 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:45:49.728 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:45:49.728 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:45:49.728 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:45:49.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:45:49.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:45:49.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:46:09.731 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:46:09.763 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:46:09.763 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:46:09.763 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:46:09.763 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:46:09.763 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:46:09.763 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:46:09.763 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:46:09.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:46:09.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:46:09.766 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:46:09.766 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:46:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:46:09.766 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:46:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:46:09.766 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:46:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:46:09.766 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:46:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:46:09.766 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:46:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:46:09.766 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:46:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:46:09.766 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:46:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:46:09.766 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:46:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:46:09.766 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:46:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:46:09.766 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:46:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:46:09.766 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:46:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:46:09.766 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:46:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:46:09.766 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:46:09.766 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:46:09.766 139328 ERROR oslo_service.periodic_task >2018-06-24 18:46:39.647 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:46:39.649 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:46:39.656 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:46:39.656 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:46:39.692 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:46:39.692 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:46:39.692 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:46:39.692 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:46:39.692 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:46:39.692 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:46:39.692 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:46:39.694 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:46:39.694 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:46:39.695 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:46:49.695 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:46:49.726 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:46:49.726 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:46:49.726 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:46:49.726 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:46:49.726 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:46:49.726 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:46:49.726 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:46:49.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:46:49.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:46:49.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:47:09.729 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:47:09.763 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:47:09.763 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:47:09.763 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:47:09.763 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:47:09.763 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:47:09.763 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:47:09.763 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:47:09.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:47:09.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:47:09.766 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:47:09.766 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:47:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:47:09.766 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:47:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:47:09.766 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:47:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:47:09.766 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:47:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:47:09.766 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:47:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:47:09.766 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:47:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:47:09.766 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:47:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:47:09.766 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:47:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:47:09.766 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:47:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:47:09.766 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:47:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:47:09.766 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:47:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:47:09.766 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:47:09.766 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:47:09.766 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:47:09.766 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:47:09.766 139328 ERROR oslo_service.periodic_task >2018-06-24 18:47:39.647 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:47:39.648 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:47:39.655 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:47:39.655 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:47:39.689 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:47:39.689 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:47:39.689 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:47:39.689 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:47:39.689 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:47:39.689 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:47:39.689 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:47:39.691 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:47:39.692 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:47:39.692 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:47:49.692 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:47:49.728 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:47:49.728 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:47:49.728 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:47:49.728 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:47:49.728 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:47:49.728 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:47:49.728 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:47:49.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:47:49.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:47:49.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:48:09.731 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:48:09.764 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:48:09.764 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:48:09.764 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:48:09.764 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:48:09.764 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:48:09.764 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:48:09.764 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:48:09.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:48:09.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:48:09.767 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:48:09.767 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:48:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:48:09.767 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:48:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:48:09.767 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:48:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:48:09.767 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:48:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:48:09.767 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:48:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:48:09.767 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:48:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:48:09.767 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:48:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:48:09.767 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:48:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:48:09.767 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:48:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:48:09.767 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:48:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:48:09.767 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:48:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:48:09.767 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:48:09.767 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:48:09.767 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:48:09.767 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:48:09.767 139328 ERROR oslo_service.periodic_task >2018-06-24 18:48:39.648 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:48:39.649 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:48:39.657 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:48:39.657 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:48:39.692 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:48:39.692 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:48:39.692 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:48:39.692 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:48:39.692 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:48:39.692 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:48:39.692 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:48:39.695 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:48:39.695 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:48:39.695 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:48:49.696 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:48:49.733 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:48:49.733 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:48:49.733 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:48:49.733 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:48:49.733 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:48:49.733 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:48:49.733 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:48:49.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:48:49.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:48:49.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:49:09.736 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:49:09.771 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:49:09.771 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:49:09.771 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:49:09.771 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:49:09.771 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:49:09.771 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:49:09.771 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:49:09.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:49:09.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:49:09.774 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:49:09.774 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:49:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:49:09.774 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:49:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:49:09.774 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:49:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:49:09.774 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:49:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:49:09.774 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:49:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:49:09.774 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:49:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:49:09.774 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:49:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:49:09.774 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:49:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:49:09.774 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:49:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:49:09.774 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:49:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:49:09.774 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:49:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:49:09.774 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:49:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:49:09.774 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:49:09.774 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:49:09.774 139328 ERROR oslo_service.periodic_task >2018-06-24 18:49:39.645 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:49:39.646 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:49:39.659 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:49:39.659 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:49:39.693 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:49:39.693 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:49:39.693 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:49:39.693 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:49:39.693 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:49:39.693 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:49:39.693 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:49:39.695 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:49:39.695 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:49:39.696 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:49:49.696 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:49:49.739 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:49:49.739 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:49:49.739 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:49:49.739 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:49:49.739 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:49:49.739 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:49:49.739 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:49:49.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:49:49.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.082 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:49:49.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:50:09.742 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:50:09.775 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:50:09.775 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:50:09.775 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:50:09.775 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:50:09.775 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:50:09.775 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:50:09.775 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:50:09.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:50:09.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:50:09.778 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:50:09.778 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:50:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:50:09.778 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:50:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:50:09.778 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:50:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:50:09.778 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:50:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:50:09.778 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:50:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:50:09.778 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:50:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:50:09.778 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:50:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:50:09.778 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:50:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:50:09.778 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:50:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:50:09.778 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:50:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:50:09.778 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:50:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:50:09.778 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:50:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:50:09.778 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:50:09.778 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:50:09.778 139328 ERROR oslo_service.periodic_task >2018-06-24 18:50:39.649 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:50:39.650 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:50:39.657 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:50:39.658 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:50:39.690 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:50:39.690 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:50:39.690 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:50:39.690 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:50:39.690 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:50:39.690 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:50:39.690 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:50:39.692 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:50:39.692 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:50:39.692 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:50:49.693 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:50:49.724 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:50:49.724 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:50:49.724 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:50:49.724 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:50:49.724 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:50:49.724 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:50:49.724 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:50:49.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:50:49.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.069 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:50:49.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:51:09.727 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:51:09.760 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:51:09.760 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:51:09.760 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:51:09.760 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:51:09.760 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:51:09.760 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:51:09.760 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:51:09.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:51:09.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.105 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:51:09.763 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:51:09.763 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:51:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:51:09.763 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:51:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:51:09.763 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:51:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:51:09.763 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:51:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:51:09.763 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:51:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:51:09.763 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:51:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:51:09.763 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:51:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:51:09.763 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:51:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:51:09.763 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:51:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:51:09.763 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:51:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:51:09.763 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:51:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:51:09.763 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:51:09.763 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:51:09.763 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:51:09.763 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:51:09.763 139328 ERROR oslo_service.periodic_task >2018-06-24 18:51:39.654 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:51:39.655 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:51:39.662 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:51:39.662 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:51:39.699 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:51:39.699 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:51:39.699 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:51:39.699 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:51:39.699 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:51:39.699 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:51:39.699 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:51:39.701 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:51:39.701 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:51:39.702 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:51:49.702 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:51:49.737 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:51:49.737 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:51:49.737 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:51:49.737 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:51:49.737 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:51:49.737 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:51:49.737 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:51:49.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:51:49.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:51:49.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:52:09.740 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:52:09.773 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:52:09.773 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:52:09.773 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:52:09.773 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:52:09.773 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:52:09.773 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:52:09.773 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:52:09.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:52:09.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:52:09.776 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:52:09.776 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:52:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:52:09.776 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:52:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:52:09.776 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:52:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:52:09.776 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:52:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:52:09.776 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:52:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:52:09.776 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:52:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:52:09.776 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:52:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:52:09.776 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:52:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:52:09.776 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:52:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:52:09.776 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:52:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:52:09.776 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:52:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:52:09.776 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:52:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:52:09.776 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:52:09.776 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:52:09.776 139328 ERROR oslo_service.periodic_task >2018-06-24 18:52:39.657 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:52:39.658 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:52:39.666 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:52:39.667 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:52:39.697 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:52:39.697 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:52:39.697 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:52:39.697 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:52:39.697 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:52:39.697 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:52:39.697 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:52:39.699 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:52:39.700 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.033 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:52:39.700 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:52:49.700 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:52:49.735 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:52:49.735 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:52:49.735 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:52:49.735 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:52:49.735 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:52:49.735 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:52:49.735 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:52:49.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:52:49.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:52:49.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:53:09.739 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:53:09.771 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:53:09.771 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:53:09.771 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:53:09.771 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:53:09.771 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:53:09.771 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:53:09.771 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:53:09.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:53:09.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.107 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:53:09.774 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:53:09.774 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:53:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:53:09.774 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:53:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:53:09.774 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:53:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:53:09.774 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:53:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:53:09.774 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:53:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:53:09.774 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:53:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:53:09.774 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:53:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:53:09.774 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:53:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:53:09.774 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:53:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:53:09.774 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:53:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:53:09.774 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:53:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:53:09.774 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:53:09.774 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:53:09.774 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:53:09.774 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:53:09.774 139328 ERROR oslo_service.periodic_task >2018-06-24 18:53:39.656 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:53:39.657 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:53:39.667 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:53:39.667 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:53:39.703 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:53:39.703 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:53:39.703 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:53:39.703 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:53:39.703 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:53:39.703 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:53:39.703 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:53:39.705 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:53:39.705 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:53:39.706 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:53:49.706 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:53:49.739 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:53:49.739 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:53:49.739 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:53:49.739 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:53:49.739 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:53:49.739 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:53:49.739 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:53:49.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:53:49.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:53:49.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:54:09.742 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:54:09.776 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:54:09.776 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:54:09.776 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:54:09.776 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:54:09.776 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:54:09.776 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:54:09.776 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:54:09.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:54:09.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:54:09.779 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:54:09.779 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:54:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:54:09.779 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:54:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:54:09.779 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:54:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:54:09.779 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:54:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:54:09.779 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:54:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:54:09.779 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:54:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:54:09.779 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:54:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:54:09.779 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:54:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:54:09.779 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:54:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:54:09.779 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:54:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:54:09.779 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:54:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:54:09.779 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:54:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:54:09.779 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:54:09.779 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:54:09.779 139328 ERROR oslo_service.periodic_task >2018-06-24 18:54:39.661 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:54:39.662 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:54:39.669 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:54:39.669 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:54:39.705 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:54:39.705 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:54:39.705 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:54:39.705 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:54:39.705 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:54:39.705 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:54:39.705 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:54:39.707 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:54:39.707 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:54:39.708 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:54:49.708 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:54:49.739 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:54:49.739 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:54:49.739 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:54:49.739 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:54:49.739 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:54:49.739 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:54:49.739 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:54:49.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:54:49.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:54:49.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:55:09.742 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:55:09.776 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:55:09.776 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:55:09.776 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:55:09.776 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:55:09.776 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:55:09.776 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:55:09.776 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:55:09.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:55:09.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:55:09.779 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:55:09.779 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:55:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:55:09.779 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:55:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:55:09.779 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:55:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:55:09.779 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:55:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:55:09.779 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:55:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:55:09.779 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:55:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:55:09.779 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:55:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:55:09.779 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:55:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:55:09.779 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:55:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:55:09.779 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:55:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:55:09.779 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:55:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:55:09.779 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:55:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:55:09.779 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:55:09.779 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:55:09.779 139328 ERROR oslo_service.periodic_task >2018-06-24 18:55:39.660 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:55:39.661 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:55:39.669 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:55:39.670 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:55:39.709 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:55:39.709 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:55:39.709 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:55:39.709 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:55:39.709 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:55:39.709 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:55:39.709 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:55:39.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:55:39.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.043 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:55:39.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:55:49.714 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:55:49.747 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:55:49.747 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:55:49.747 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:55:49.747 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:55:49.747 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:55:49.747 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:55:49.747 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:55:49.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:55:49.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.08 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:55:49.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:56:09.751 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:56:09.784 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:56:09.784 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:56:09.784 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:56:09.784 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:56:09.784 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:56:09.784 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:56:09.784 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:56:09.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:56:09.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:56:09.787 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:56:09.787 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:56:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:56:09.787 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:56:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:56:09.787 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:56:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:56:09.787 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:56:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:56:09.787 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:56:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:56:09.787 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:56:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:56:09.787 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:56:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:56:09.787 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:56:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:56:09.787 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:56:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:56:09.787 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:56:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:56:09.787 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:56:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:56:09.787 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:56:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:56:09.787 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:56:09.787 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:56:09.787 139328 ERROR oslo_service.periodic_task >2018-06-24 18:56:39.658 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:56:39.660 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:56:39.667 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:56:39.667 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:56:39.702 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:56:39.702 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:56:39.702 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:56:39.702 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:56:39.702 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:56:39.702 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:56:39.702 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:56:39.704 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:56:39.704 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:56:39.704 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:56:49.705 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:56:49.738 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:56:49.738 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:56:49.738 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:56:49.738 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:56:49.738 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:56:49.738 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:56:49.738 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:56:49.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:56:49.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:56:49.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:57:09.741 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:57:09.778 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:57:09.778 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:57:09.778 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:57:09.778 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:57:09.778 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:57:09.778 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:57:09.778 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:57:09.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:57:09.781 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:57:09.781 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:57:09.781 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:57:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:57:09.781 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:57:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:57:09.781 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:57:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:57:09.781 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:57:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:57:09.781 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:57:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:57:09.781 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:57:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:57:09.781 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:57:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:57:09.781 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:57:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:57:09.781 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:57:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:57:09.781 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:57:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:57:09.781 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:57:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:57:09.781 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:57:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:57:09.781 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:57:09.781 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:57:09.781 139328 ERROR oslo_service.periodic_task >2018-06-24 18:57:39.663 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:57:39.664 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:57:39.698 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:57:39.698 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:57:39.698 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:57:39.698 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:57:39.698 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:57:39.698 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:57:39.698 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:57:39.701 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:57:39.701 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:57:39.702 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:57:49.702 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:57:49.739 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:57:49.739 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:57:49.739 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:57:49.739 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:57:49.739 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:57:49.739 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:57:49.739 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:57:49.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:57:49.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:57:49.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:58:09.743 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:58:09.774 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:58:09.774 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:58:09.774 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:58:09.774 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:58:09.774 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:58:09.774 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:58:09.774 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:58:09.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:58:09.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:58:09.776 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:58:09.776 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:58:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:58:09.776 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:58:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:58:09.776 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:58:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:58:09.776 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:58:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:58:09.776 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:58:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:58:09.776 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:58:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:58:09.776 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:58:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:58:09.776 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:58:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:58:09.776 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:58:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:58:09.776 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:58:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:58:09.776 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:58:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:58:09.776 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:58:09.776 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:58:09.776 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:58:09.776 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:58:09.776 139328 ERROR oslo_service.periodic_task >2018-06-24 18:58:39.668 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:58:39.669 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:58:39.676 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:58:39.677 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:58:39.709 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:58:39.709 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:58:39.709 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:58:39.709 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:58:39.709 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:58:39.709 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:58:39.709 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:58:39.711 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:58:39.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:58:39.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:58:49.713 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:58:49.765 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:58:49.765 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:58:49.765 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:58:49.765 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:58:49.765 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:58:49.765 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:58:49.765 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:58:49.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:58:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.091 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:58:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:59:09.768 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:59:09.806 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:59:09.806 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:59:09.806 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:59:09.806 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:59:09.806 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:59:09.806 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:59:09.806 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:59:09.809 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:59:09.809 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.132 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:59:09.810 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:59:09.810 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 18:59:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 18:59:09.810 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 18:59:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 18:59:09.810 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 18:59:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 18:59:09.810 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 18:59:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 18:59:09.810 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 18:59:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 18:59:09.810 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 18:59:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 18:59:09.810 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 18:59:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 18:59:09.810 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 18:59:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 18:59:09.810 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 18:59:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 18:59:09.810 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 18:59:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 18:59:09.810 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 18:59:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 18:59:09.810 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 18:59:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 18:59:09.810 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 18:59:09.810 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 18:59:09.810 139328 ERROR oslo_service.periodic_task >2018-06-24 18:59:39.671 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:59:39.672 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 18:59:39.681 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 18:59:39.681 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:59:39.716 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:59:39.716 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:59:39.716 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:59:39.716 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:59:39.716 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:59:39.716 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:59:39.716 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:59:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:59:39.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:59:39.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 18:59:49.720 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 18:59:49.755 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 18:59:49.755 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 18:59:49.755 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 18:59:49.755 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 18:59:49.755 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 18:59:49.755 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 18:59:49.755 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 18:59:49.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 18:59:49.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 18:59:49.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:00:09.759 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:00:09.797 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:00:09.797 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:00:09.797 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:00:09.797 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:00:09.797 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:00:09.797 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:00:09.797 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:00:09.799 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:00:09.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.118 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:00:09.800 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:00:09.800 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:00:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:00:09.800 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:00:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:00:09.800 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:00:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:00:09.800 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:00:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:00:09.800 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:00:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:00:09.800 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:00:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:00:09.800 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:00:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:00:09.800 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:00:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:00:09.800 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:00:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:00:09.800 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:00:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:00:09.800 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:00:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:00:09.800 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:00:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:00:09.800 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:00:09.800 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:00:09.800 139328 ERROR oslo_service.periodic_task >2018-06-24 19:00:39.671 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:00:39.673 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:00:39.679 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:00:39.680 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:00:39.712 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:00:39.712 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:00:39.712 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:00:39.712 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:00:39.712 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:00:39.712 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:00:39.712 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:00:39.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:00:39.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:00:39.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:00:49.715 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:00:49.753 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:00:49.753 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:00:49.753 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:00:49.753 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:00:49.753 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:00:49.753 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:00:49.753 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:00:49.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:00:49.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:00:49.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:01:09.757 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:01:09.798 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:01:09.798 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:01:09.798 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:01:09.798 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:01:09.798 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:01:09.798 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:01:09.798 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:01:09.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:01:09.801 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.121 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:01:09.801 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:01:09.801 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:01:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:01:09.801 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:01:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:01:09.801 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:01:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:01:09.801 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:01:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:01:09.801 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:01:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:01:09.801 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:01:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:01:09.801 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:01:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:01:09.801 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:01:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:01:09.801 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:01:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:01:09.801 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:01:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:01:09.801 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:01:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:01:09.801 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:01:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:01:09.801 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:01:09.801 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:01:09.801 139328 ERROR oslo_service.periodic_task >2018-06-24 19:01:39.673 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:01:39.674 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:01:39.682 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:01:39.683 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:01:39.722 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:01:39.722 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:01:39.722 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:01:39.722 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:01:39.722 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:01:39.722 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:01:39.722 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:01:39.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:01:39.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.042 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:01:39.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:01:49.726 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:01:49.760 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:01:49.760 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:01:49.760 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:01:49.760 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:01:49.760 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:01:49.760 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:01:49.760 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:01:49.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:01:49.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:01:49.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:02:09.763 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:02:09.794 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:02:09.794 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:02:09.794 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:02:09.794 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:02:09.794 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:02:09.794 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:02:09.794 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:02:09.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:02:09.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:02:09.797 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:02:09.797 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:02:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:02:09.797 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:02:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:02:09.797 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:02:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:02:09.797 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:02:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:02:09.797 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:02:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:02:09.797 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:02:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:02:09.797 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:02:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:02:09.797 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:02:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:02:09.797 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:02:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:02:09.797 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:02:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:02:09.797 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:02:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:02:09.797 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:02:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:02:09.797 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:02:09.797 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:02:09.797 139328 ERROR oslo_service.periodic_task >2018-06-24 19:02:39.668 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:02:39.669 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:02:39.676 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:02:39.677 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:02:39.715 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:02:39.715 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:02:39.715 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:02:39.715 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:02:39.715 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:02:39.715 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:02:39.715 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:02:39.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:02:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:02:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:02:49.718 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:02:49.750 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:02:49.750 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:02:49.750 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:02:49.750 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:02:49.750 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:02:49.750 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:02:49.750 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:02:49.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:02:49.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:02:49.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:03:09.753 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:03:09.787 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:03:09.787 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:03:09.787 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:03:09.787 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:03:09.787 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:03:09.787 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:03:09.787 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:03:09.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:03:09.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:03:09.790 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:03:09.790 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:03:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:03:09.790 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:03:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:03:09.790 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:03:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:03:09.790 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:03:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:03:09.790 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:03:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:03:09.790 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:03:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:03:09.790 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:03:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:03:09.790 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:03:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:03:09.790 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:03:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:03:09.790 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:03:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:03:09.790 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:03:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:03:09.790 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:03:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:03:09.790 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:03:09.790 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:03:09.790 139328 ERROR oslo_service.periodic_task >2018-06-24 19:03:39.671 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:03:39.672 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:03:39.678 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:03:39.679 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:03:39.715 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:03:39.715 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:03:39.715 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:03:39.715 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:03:39.715 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:03:39.715 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:03:39.715 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:03:39.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:03:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:03:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:03:49.720 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:03:49.769 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:03:49.769 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:03:49.769 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:03:49.769 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:03:49.769 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:03:49.769 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:03:49.769 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:03:49.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:03:49.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.093 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:03:49.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:04:09.773 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:04:09.806 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:04:09.806 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:04:09.806 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:04:09.806 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:04:09.806 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:04:09.806 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:04:09.806 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:04:09.808 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:04:09.808 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.13 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:04:09.809 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:04:09.809 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:04:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:04:09.809 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:04:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:04:09.809 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:04:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:04:09.809 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:04:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:04:09.809 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:04:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:04:09.809 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:04:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:04:09.809 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:04:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:04:09.809 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:04:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:04:09.809 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:04:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:04:09.809 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:04:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:04:09.809 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:04:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:04:09.809 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:04:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:04:09.809 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:04:09.809 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:04:09.809 139328 ERROR oslo_service.periodic_task >2018-06-24 19:04:39.670 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:04:39.672 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:04:39.678 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:04:39.679 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:04:39.712 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:04:39.712 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:04:39.712 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:04:39.712 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:04:39.712 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:04:39.712 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:04:39.712 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:04:39.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:04:39.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:04:39.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:04:49.716 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:04:49.751 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:04:49.751 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:04:49.751 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:04:49.751 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:04:49.751 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:04:49.751 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:04:49.751 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:04:49.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:04:49.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:04:49.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:05:09.754 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:05:09.789 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:05:09.789 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:05:09.789 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:05:09.789 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:05:09.789 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:05:09.789 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:05:09.789 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:05:09.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:05:09.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:05:09.792 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:05:09.792 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:05:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:05:09.792 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:05:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:05:09.792 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:05:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:05:09.792 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:05:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:05:09.792 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:05:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:05:09.792 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:05:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:05:09.792 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:05:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:05:09.792 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:05:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:05:09.792 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:05:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:05:09.792 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:05:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:05:09.792 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:05:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:05:09.792 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:05:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:05:09.792 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:05:09.792 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:05:09.792 139328 ERROR oslo_service.periodic_task >2018-06-24 19:05:39.673 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:05:39.674 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:05:39.685 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:05:39.686 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:05:39.724 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:05:39.724 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:05:39.724 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:05:39.724 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:05:39.724 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:05:39.724 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:05:39.724 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:05:39.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:05:39.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.041 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:05:39.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:05:49.728 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:05:49.762 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:05:49.762 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:05:49.762 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:05:49.762 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:05:49.762 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:05:49.762 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:05:49.762 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:05:49.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:05:49.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.08 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:05:49.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:06:09.766 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:06:09.802 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:06:09.802 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:06:09.802 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:06:09.802 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:06:09.802 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:06:09.802 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:06:09.802 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:06:09.804 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:06:09.805 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.119 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:06:09.805 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:06:09.805 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:06:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:06:09.805 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:06:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:06:09.805 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:06:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:06:09.805 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:06:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:06:09.805 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:06:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:06:09.805 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:06:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:06:09.805 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:06:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:06:09.805 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:06:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:06:09.805 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:06:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:06:09.805 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:06:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:06:09.805 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:06:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:06:09.805 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:06:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:06:09.805 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:06:09.805 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:06:09.805 139328 ERROR oslo_service.periodic_task >2018-06-24 19:06:39.676 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:06:39.677 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:06:39.687 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:06:39.687 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:06:39.721 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:06:39.721 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:06:39.721 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:06:39.721 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:06:39.721 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:06:39.721 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:06:39.721 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:06:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:06:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:06:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:06:49.725 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:06:49.759 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:06:49.759 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:06:49.759 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:06:49.759 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:06:49.759 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:06:49.759 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:06:49.759 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:06:49.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:06:49.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:06:49.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:07:09.762 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:07:09.795 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:07:09.795 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:07:09.795 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:07:09.795 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:07:09.795 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:07:09.795 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:07:09.795 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:07:09.797 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:07:09.797 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:07:09.797 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:07:09.797 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:07:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:07:09.797 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:07:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:07:09.797 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:07:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:07:09.797 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:07:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:07:09.797 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:07:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:07:09.797 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:07:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:07:09.797 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:07:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:07:09.797 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:07:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:07:09.797 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:07:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:07:09.797 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:07:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:07:09.797 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:07:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:07:09.797 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:07:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:07:09.797 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:07:09.797 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:07:09.797 139328 ERROR oslo_service.periodic_task >2018-06-24 19:07:39.679 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:07:39.680 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:07:39.688 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:07:39.688 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:07:39.723 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:07:39.723 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:07:39.723 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:07:39.723 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:07:39.723 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:07:39.723 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:07:39.723 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:07:39.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:07:39.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:07:39.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:07:49.726 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:07:49.767 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:07:49.767 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:07:49.767 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:07:49.767 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:07:49.767 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:07:49.767 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:07:49.767 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:07:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:07:49.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.081 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:07:49.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:08:09.771 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:08:09.807 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:08:09.807 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:08:09.807 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:08:09.807 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:08:09.807 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:08:09.807 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:08:09.807 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:08:09.810 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:08:09.810 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.121 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:08:09.810 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:08:09.810 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:08:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:08:09.810 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:08:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:08:09.810 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:08:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:08:09.810 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:08:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:08:09.810 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:08:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:08:09.810 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:08:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:08:09.810 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:08:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:08:09.810 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:08:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:08:09.810 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:08:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:08:09.810 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:08:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:08:09.810 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:08:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:08:09.810 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:08:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:08:09.810 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:08:09.810 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:08:09.810 139328 ERROR oslo_service.periodic_task >2018-06-24 19:08:39.682 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:08:39.682 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:08:39.690 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:08:39.690 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:08:39.722 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:08:39.722 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:08:39.722 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:08:39.722 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:08:39.722 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:08:39.722 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:08:39.722 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:08:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:08:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:08:39.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:08:49.725 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:08:49.757 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:08:49.757 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:08:49.757 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:08:49.757 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:08:49.757 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:08:49.757 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:08:49.757 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:08:49.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:08:49.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.07 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:08:49.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:09:09.762 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:09:09.795 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:09:09.795 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:09:09.795 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:09:09.795 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:09:09.795 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:09:09.795 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:09:09.795 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:09:09.797 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:09:09.797 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.106 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:09:09.797 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:09:09.797 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:09:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:09:09.797 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:09:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:09:09.797 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:09:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:09:09.797 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:09:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:09:09.797 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:09:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:09:09.797 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:09:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:09:09.797 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:09:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:09:09.797 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:09:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:09:09.797 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:09:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:09:09.797 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:09:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:09:09.797 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:09:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:09:09.797 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:09:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:09:09.797 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:09:09.797 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:09:09.797 139328 ERROR oslo_service.periodic_task >2018-06-24 19:09:39.679 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:09:39.680 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:09:39.686 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:09:39.687 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:09:39.719 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:09:39.719 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:09:39.719 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:09:39.719 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:09:39.719 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:09:39.719 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:09:39.719 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:09:39.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:09:39.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:09:39.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:09:49.722 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:09:49.754 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:09:49.754 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:09:49.754 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:09:49.754 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:09:49.754 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:09:49.754 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:09:49.754 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:09:49.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:09:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.07 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:09:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:10:09.757 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:10:09.792 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:10:09.792 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:10:09.792 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:10:09.792 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:10:09.792 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:10:09.792 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:10:09.792 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:10:09.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:10:09.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.107 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:10:09.794 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:10:09.794 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:10:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:10:09.794 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:10:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:10:09.794 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:10:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:10:09.794 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:10:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:10:09.794 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:10:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:10:09.794 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:10:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:10:09.794 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:10:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:10:09.794 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:10:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:10:09.794 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:10:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:10:09.794 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:10:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:10:09.794 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:10:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:10:09.794 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:10:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:10:09.794 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:10:09.794 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:10:09.794 139328 ERROR oslo_service.periodic_task >2018-06-24 19:10:39.676 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:10:39.677 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:10:39.685 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:10:39.686 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:10:39.721 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:10:39.721 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:10:39.721 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:10:39.721 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:10:39.721 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:10:39.721 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:10:39.721 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:10:39.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:10:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:10:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:10:49.725 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:10:49.757 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:10:49.757 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:10:49.757 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:10:49.757 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:10:49.757 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:10:49.757 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:10:49.757 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:10:49.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:10:49.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:10:49.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:11:09.760 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:11:09.799 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:11:09.799 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:11:09.799 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:11:09.799 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:11:09.799 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:11:09.799 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:11:09.799 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:11:09.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:11:09.803 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:11:09.803 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:11:09.803 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:11:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:11:09.803 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:11:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:11:09.803 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:11:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:11:09.803 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:11:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:11:09.803 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:11:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:11:09.803 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:11:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:11:09.803 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:11:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:11:09.803 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:11:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:11:09.803 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:11:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:11:09.803 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:11:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:11:09.803 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:11:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:11:09.803 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:11:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:11:09.803 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:11:09.803 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:11:09.803 139328 ERROR oslo_service.periodic_task >2018-06-24 19:11:39.674 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:11:39.675 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:11:39.685 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:11:39.685 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:11:39.720 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:11:39.720 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:11:39.720 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:11:39.720 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:11:39.720 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:11:39.720 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:11:39.720 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:11:39.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:11:39.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:11:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:11:49.724 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:11:49.758 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:11:49.758 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:11:49.758 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:11:49.758 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:11:49.758 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:11:49.758 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:11:49.758 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:11:49.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:11:49.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:11:49.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:12:09.761 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:12:09.795 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:12:09.795 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:12:09.795 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:12:09.795 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:12:09.795 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:12:09.795 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:12:09.795 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:12:09.797 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:12:09.798 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:12:09.798 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:12:09.798 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:12:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:12:09.798 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:12:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:12:09.798 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:12:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:12:09.798 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:12:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:12:09.798 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:12:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:12:09.798 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:12:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:12:09.798 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:12:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:12:09.798 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:12:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:12:09.798 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:12:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:12:09.798 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:12:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:12:09.798 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:12:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:12:09.798 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:12:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:12:09.798 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:12:09.798 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:12:09.798 139328 ERROR oslo_service.periodic_task >2018-06-24 19:12:39.669 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:12:39.670 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:12:39.678 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:12:39.679 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:12:39.716 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:12:39.716 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:12:39.716 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:12:39.716 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:12:39.716 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:12:39.716 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:12:39.716 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:12:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:12:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:12:39.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:12:49.719 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:12:49.751 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:12:49.751 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:12:49.751 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:12:49.751 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:12:49.751 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:12:49.751 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:12:49.751 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:12:49.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:12:49.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:12:49.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:13:09.754 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:13:09.790 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:13:09.790 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:13:09.790 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:13:09.790 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:13:09.790 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:13:09.790 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:13:09.790 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:13:09.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:13:09.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:13:09.793 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:13:09.793 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:13:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:13:09.793 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:13:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:13:09.793 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:13:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:13:09.793 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:13:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:13:09.793 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:13:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:13:09.793 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:13:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:13:09.793 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:13:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:13:09.793 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:13:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:13:09.793 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:13:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:13:09.793 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:13:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:13:09.793 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:13:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:13:09.793 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:13:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:13:09.793 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:13:09.793 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:13:09.793 139328 ERROR oslo_service.periodic_task >2018-06-24 19:13:39.674 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:13:39.675 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:13:39.689 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:13:39.689 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:13:39.725 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:13:39.725 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:13:39.725 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:13:39.725 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:13:39.725 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:13:39.725 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:13:39.725 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:13:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:13:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:13:39.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:13:49.729 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:13:49.764 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:13:49.764 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:13:49.764 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:13:49.764 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:13:49.764 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:13:49.764 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:13:49.764 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:13:49.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:13:49.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:13:49.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:14:09.767 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:14:09.802 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:14:09.802 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:14:09.802 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:14:09.802 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:14:09.802 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:14:09.802 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:14:09.802 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:14:09.804 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:14:09.804 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:14:09.805 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:14:09.805 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:14:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:14:09.805 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:14:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:14:09.805 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:14:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:14:09.805 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:14:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:14:09.805 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:14:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:14:09.805 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:14:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:14:09.805 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:14:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:14:09.805 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:14:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:14:09.805 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:14:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:14:09.805 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:14:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:14:09.805 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:14:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:14:09.805 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:14:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:14:09.805 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:14:09.805 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:14:09.805 139328 ERROR oslo_service.periodic_task >2018-06-24 19:14:39.676 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:14:39.677 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:14:39.684 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:14:39.685 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:14:39.724 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:14:39.724 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:14:39.724 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:14:39.724 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:14:39.724 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:14:39.724 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:14:39.724 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:14:39.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:14:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.042 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:14:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:14:49.729 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:14:49.765 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:14:49.765 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:14:49.765 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:14:49.765 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:14:49.765 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:14:49.765 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:14:49.765 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:14:49.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:14:49.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.081 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:14:49.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:15:09.767 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:15:09.804 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:15:09.804 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:15:09.804 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:15:09.804 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:15:09.804 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:15:09.804 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:15:09.804 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:15:09.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:15:09.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.12 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:15:09.806 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:15:09.806 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:15:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:15:09.806 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:15:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:15:09.806 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:15:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:15:09.806 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:15:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:15:09.806 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:15:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:15:09.806 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:15:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:15:09.806 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:15:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:15:09.806 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:15:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:15:09.806 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:15:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:15:09.806 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:15:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:15:09.806 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:15:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:15:09.806 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:15:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:15:09.806 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:15:09.806 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:15:09.806 139328 ERROR oslo_service.periodic_task >2018-06-24 19:15:39.678 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:15:39.679 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:15:39.686 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:15:39.687 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:15:39.720 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:15:39.720 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:15:39.720 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:15:39.720 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:15:39.720 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:15:39.720 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:15:39.720 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:15:39.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:15:39.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:15:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:15:49.724 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:15:49.766 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:15:49.766 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:15:49.766 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:15:49.766 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:15:49.766 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:15:49.766 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:15:49.766 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:15:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:15:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.082 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:15:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:16:09.770 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:16:09.804 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:16:09.804 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:16:09.804 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:16:09.804 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:16:09.804 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:16:09.804 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:16:09.804 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:16:09.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:16:09.807 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.12 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:16:09.807 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:16:09.807 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:16:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:16:09.807 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:16:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:16:09.807 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:16:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:16:09.807 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:16:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:16:09.807 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:16:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:16:09.807 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:16:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:16:09.807 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:16:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:16:09.807 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:16:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:16:09.807 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:16:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:16:09.807 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:16:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:16:09.807 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:16:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:16:09.807 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:16:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:16:09.807 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:16:09.807 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:16:09.807 139328 ERROR oslo_service.periodic_task >2018-06-24 19:16:39.678 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:16:39.680 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:16:39.694 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:16:39.694 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:16:39.728 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:16:39.728 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:16:39.728 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:16:39.728 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:16:39.728 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:16:39.728 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:16:39.728 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:16:39.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:16:39.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:16:39.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:16:49.732 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:16:49.764 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:16:49.764 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:16:49.764 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:16:49.764 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:16:49.764 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:16:49.764 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:16:49.764 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:16:49.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:16:49.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:16:49.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:17:09.767 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:17:09.800 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:17:09.800 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:17:09.800 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:17:09.800 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:17:09.800 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:17:09.800 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:17:09.800 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:17:09.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:17:09.803 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:17:09.803 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:17:09.803 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:17:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:17:09.803 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:17:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:17:09.803 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:17:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:17:09.803 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:17:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:17:09.803 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:17:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:17:09.803 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:17:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:17:09.803 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:17:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:17:09.803 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:17:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:17:09.803 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:17:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:17:09.803 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:17:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:17:09.803 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:17:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:17:09.803 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:17:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:17:09.803 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:17:09.803 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:17:09.803 139328 ERROR oslo_service.periodic_task >2018-06-24 19:17:39.674 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:17:39.675 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:17:39.682 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:17:39.682 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:17:39.715 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:17:39.715 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:17:39.715 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:17:39.715 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:17:39.715 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:17:39.715 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:17:39.715 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:17:39.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:17:39.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:17:39.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:17:49.718 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:17:49.751 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:17:49.751 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:17:49.751 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:17:49.751 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:17:49.751 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:17:49.751 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:17:49.751 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:17:49.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:17:49.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:17:49.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:18:09.755 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:18:09.790 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:18:09.790 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:18:09.790 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:18:09.790 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:18:09.790 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:18:09.790 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:18:09.790 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:18:09.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:18:09.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:18:09.792 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:18:09.792 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:18:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:18:09.792 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:18:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:18:09.792 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:18:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:18:09.792 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:18:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:18:09.792 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:18:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:18:09.792 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:18:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:18:09.792 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:18:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:18:09.792 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:18:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:18:09.792 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:18:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:18:09.792 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:18:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:18:09.792 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:18:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:18:09.792 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:18:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:18:09.792 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:18:09.792 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:18:09.792 139328 ERROR oslo_service.periodic_task >2018-06-24 19:18:39.676 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:18:39.677 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:18:39.684 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:18:39.684 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:18:39.724 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:18:39.724 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:18:39.724 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:18:39.724 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:18:39.724 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:18:39.724 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:18:39.724 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:18:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:18:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.043 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:18:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:18:49.729 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:18:49.760 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:18:49.760 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:18:49.760 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:18:49.760 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:18:49.760 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:18:49.760 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:18:49.760 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:18:49.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:18:49.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:18:49.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:19:09.763 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:19:09.805 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:19:09.805 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:19:09.805 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:19:09.805 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:19:09.805 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:19:09.805 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:19:09.805 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:19:09.807 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:19:09.808 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.123 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:19:09.808 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:19:09.808 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:19:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:19:09.808 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:19:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:19:09.808 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:19:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:19:09.808 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:19:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:19:09.808 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:19:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:19:09.808 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:19:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:19:09.808 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:19:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:19:09.808 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:19:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:19:09.808 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:19:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:19:09.808 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:19:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:19:09.808 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:19:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:19:09.808 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:19:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:19:09.808 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:19:09.808 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:19:09.808 139328 ERROR oslo_service.periodic_task >2018-06-24 19:19:39.680 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:19:39.681 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:19:39.687 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:19:39.688 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:19:39.721 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:19:39.721 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:19:39.721 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:19:39.721 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:19:39.721 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:19:39.721 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:19:39.721 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:19:39.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:19:39.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:19:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:19:49.724 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:19:49.756 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:19:49.756 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:19:49.756 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:19:49.756 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:19:49.756 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:19:49.756 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:19:49.756 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:19:49.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:19:49.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.07 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:19:49.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:20:09.759 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:20:09.792 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:20:09.792 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:20:09.792 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:20:09.792 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:20:09.792 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:20:09.792 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:20:09.792 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:20:09.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:20:09.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.106 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:20:09.794 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:20:09.794 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:20:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:20:09.794 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:20:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:20:09.794 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:20:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:20:09.794 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:20:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:20:09.794 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:20:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:20:09.794 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:20:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:20:09.794 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:20:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:20:09.794 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:20:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:20:09.794 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:20:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:20:09.794 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:20:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:20:09.794 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:20:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:20:09.794 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:20:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:20:09.794 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:20:09.794 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:20:09.794 139328 ERROR oslo_service.periodic_task >2018-06-24 19:20:39.676 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:20:39.677 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:20:39.685 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:20:39.685 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:20:39.718 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:20:39.718 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:20:39.718 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:20:39.718 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:20:39.718 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:20:39.718 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:20:39.718 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:20:39.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:20:39.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:20:39.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:20:49.721 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:20:49.755 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:20:49.755 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:20:49.755 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:20:49.755 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:20:49.755 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:20:49.755 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:20:49.755 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:20:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:20:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:20:49.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:21:09.758 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:21:09.792 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:21:09.792 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:21:09.792 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:21:09.792 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:21:09.792 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:21:09.792 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:21:09.792 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:21:09.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:21:09.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:21:09.795 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:21:09.795 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:21:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:21:09.795 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:21:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:21:09.795 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:21:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:21:09.795 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:21:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:21:09.795 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:21:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:21:09.795 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:21:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:21:09.795 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:21:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:21:09.795 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:21:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:21:09.795 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:21:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:21:09.795 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:21:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:21:09.795 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:21:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:21:09.795 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:21:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:21:09.795 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:21:09.795 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:21:09.795 139328 ERROR oslo_service.periodic_task >2018-06-24 19:21:39.676 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:21:39.677 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:21:39.683 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:21:39.683 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:21:39.716 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:21:39.716 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:21:39.716 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:21:39.716 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:21:39.716 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:21:39.716 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:21:39.716 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:21:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:21:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:21:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:21:49.719 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:21:49.752 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:21:49.752 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:21:49.752 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:21:49.752 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:21:49.752 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:21:49.752 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:21:49.752 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:21:49.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:21:49.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:21:49.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:22:09.756 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:22:09.789 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:22:09.789 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:22:09.789 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:22:09.789 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:22:09.789 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:22:09.789 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:22:09.789 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:22:09.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:22:09.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:22:09.792 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:22:09.792 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:22:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:22:09.792 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:22:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:22:09.792 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:22:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:22:09.792 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:22:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:22:09.792 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:22:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:22:09.792 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:22:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:22:09.792 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:22:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:22:09.792 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:22:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:22:09.792 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:22:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:22:09.792 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:22:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:22:09.792 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:22:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:22:09.792 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:22:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:22:09.792 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:22:09.792 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:22:09.792 139328 ERROR oslo_service.periodic_task >2018-06-24 19:22:39.674 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:22:39.675 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:22:39.682 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:22:39.682 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:22:39.713 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:22:39.713 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:22:39.713 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:22:39.713 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:22:39.713 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:22:39.713 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:22:39.713 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:22:39.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:22:39.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.033 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:22:39.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:22:49.717 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:22:49.748 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:22:49.748 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:22:49.748 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:22:49.748 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:22:49.748 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:22:49.748 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:22:49.748 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:22:49.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:22:49.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.067 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:22:49.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:23:09.751 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:23:09.784 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:23:09.784 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:23:09.784 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:23:09.784 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:23:09.784 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:23:09.784 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:23:09.784 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:23:09.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:23:09.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.104 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:23:09.787 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:23:09.787 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:23:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:23:09.787 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:23:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:23:09.787 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:23:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:23:09.787 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:23:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:23:09.787 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:23:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:23:09.787 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:23:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:23:09.787 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:23:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:23:09.787 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:23:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:23:09.787 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:23:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:23:09.787 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:23:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:23:09.787 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:23:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:23:09.787 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:23:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:23:09.787 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:23:09.787 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:23:09.787 139328 ERROR oslo_service.periodic_task >2018-06-24 19:23:39.669 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:23:39.670 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:23:39.677 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:23:39.678 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:23:39.716 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:23:39.716 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:23:39.716 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:23:39.716 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:23:39.716 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:23:39.716 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:23:39.716 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:23:39.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:23:39.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.041 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:23:39.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:23:49.720 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:23:49.752 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:23:49.752 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:23:49.752 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:23:49.752 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:23:49.752 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:23:49.752 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:23:49.752 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:23:49.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:23:49.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:23:49.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:24:09.755 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:24:09.793 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:24:09.793 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:24:09.793 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:24:09.793 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:24:09.793 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:24:09.793 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:24:09.793 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:24:09.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:24:09.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.118 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:24:09.797 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:24:09.797 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:24:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:24:09.797 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:24:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:24:09.797 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:24:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:24:09.797 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:24:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:24:09.797 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:24:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:24:09.797 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:24:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:24:09.797 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:24:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:24:09.797 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:24:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:24:09.797 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:24:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:24:09.797 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:24:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:24:09.797 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:24:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:24:09.797 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:24:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:24:09.797 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:24:09.797 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:24:09.797 139328 ERROR oslo_service.periodic_task >2018-06-24 19:24:39.669 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:24:39.670 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:24:39.676 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:24:39.677 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:24:39.710 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:24:39.710 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:24:39.710 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:24:39.710 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:24:39.710 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:24:39.710 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:24:39.710 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:24:39.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:24:39.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:24:39.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:24:49.714 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:24:49.747 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:24:49.747 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:24:49.747 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:24:49.747 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:24:49.747 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:24:49.747 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:24:49.747 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:24:49.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:24:49.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:24:49.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:25:09.750 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:25:09.788 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:25:09.788 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:25:09.788 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:25:09.788 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:25:09.788 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:25:09.788 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:25:09.788 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:25:09.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:25:09.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:25:09.791 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:25:09.791 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:25:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:25:09.791 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:25:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:25:09.791 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:25:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:25:09.791 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:25:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:25:09.791 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:25:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:25:09.791 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:25:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:25:09.791 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:25:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:25:09.791 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:25:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:25:09.791 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:25:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:25:09.791 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:25:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:25:09.791 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:25:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:25:09.791 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:25:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:25:09.791 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:25:09.791 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:25:09.791 139328 ERROR oslo_service.periodic_task >2018-06-24 19:25:39.672 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:25:39.673 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:25:39.679 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:25:39.680 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:25:39.712 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:25:39.712 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:25:39.712 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:25:39.712 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:25:39.712 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:25:39.712 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:25:39.712 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:25:39.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:25:39.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:25:39.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:25:49.716 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:25:49.748 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:25:49.748 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:25:49.748 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:25:49.748 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:25:49.748 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:25:49.748 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:25:49.748 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:25:49.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:25:49.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:25:49.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:26:09.752 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:26:09.785 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:26:09.785 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:26:09.785 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:26:09.785 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:26:09.785 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:26:09.785 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:26:09.785 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:26:09.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:26:09.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:26:09.789 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:26:09.789 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:26:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:26:09.789 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:26:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:26:09.789 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:26:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:26:09.789 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:26:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:26:09.789 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:26:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:26:09.789 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:26:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:26:09.789 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:26:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:26:09.789 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:26:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:26:09.789 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:26:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:26:09.789 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:26:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:26:09.789 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:26:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:26:09.789 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:26:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:26:09.789 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:26:09.789 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:26:09.789 139328 ERROR oslo_service.periodic_task >2018-06-24 19:26:39.670 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:26:39.672 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:26:39.683 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:26:39.684 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:26:39.716 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:26:39.716 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:26:39.716 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:26:39.716 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:26:39.716 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:26:39.716 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:26:39.716 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:26:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:26:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:26:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:26:49.719 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:26:49.752 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:26:49.752 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:26:49.752 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:26:49.752 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:26:49.752 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:26:49.752 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:26:49.752 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:26:49.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:26:49.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.07 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:26:49.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:27:09.755 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:27:09.790 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:27:09.790 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:27:09.790 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:27:09.790 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:27:09.790 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:27:09.790 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:27:09.790 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:27:09.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:27:09.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:27:09.793 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:27:09.793 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:27:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:27:09.793 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:27:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:27:09.793 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:27:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:27:09.793 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:27:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:27:09.793 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:27:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:27:09.793 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:27:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:27:09.793 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:27:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:27:09.793 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:27:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:27:09.793 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:27:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:27:09.793 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:27:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:27:09.793 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:27:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:27:09.793 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:27:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:27:09.793 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:27:09.793 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:27:09.793 139328 ERROR oslo_service.periodic_task >2018-06-24 19:27:39.674 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:27:39.675 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:27:39.682 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:27:39.683 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:27:39.716 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:27:39.716 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:27:39.716 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:27:39.716 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:27:39.716 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:27:39.716 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:27:39.716 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:27:39.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:27:39.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:27:39.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:27:49.720 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:27:49.753 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:27:49.753 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:27:49.753 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:27:49.753 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:27:49.753 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:27:49.753 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:27:49.753 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:27:49.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:27:49.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:27:49.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:28:09.756 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:28:09.788 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:28:09.788 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:28:09.788 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:28:09.788 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:28:09.788 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:28:09.788 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:28:09.788 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:28:09.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:28:09.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:28:09.791 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:28:09.791 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:28:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:28:09.791 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:28:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:28:09.791 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:28:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:28:09.791 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:28:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:28:09.791 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:28:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:28:09.791 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:28:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:28:09.791 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:28:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:28:09.791 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:28:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:28:09.791 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:28:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:28:09.791 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:28:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:28:09.791 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:28:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:28:09.791 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:28:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:28:09.791 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:28:09.791 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:28:09.791 139328 ERROR oslo_service.periodic_task >2018-06-24 19:28:39.672 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:28:39.673 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:28:39.679 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:28:39.680 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:28:39.714 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:28:39.714 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:28:39.714 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:28:39.714 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:28:39.714 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:28:39.714 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:28:39.714 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:28:39.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:28:39.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:28:39.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:28:49.718 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:28:49.753 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:28:49.753 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:28:49.753 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:28:49.753 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:28:49.753 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:28:49.753 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:28:49.753 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:28:49.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:28:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:28:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:29:09.758 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:29:09.792 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:29:09.792 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:29:09.792 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:29:09.792 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:29:09.792 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:29:09.792 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:29:09.792 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:29:09.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:29:09.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:29:09.795 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:29:09.795 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:29:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:29:09.795 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:29:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:29:09.795 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:29:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:29:09.795 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:29:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:29:09.795 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:29:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:29:09.795 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:29:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:29:09.795 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:29:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:29:09.795 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:29:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:29:09.795 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:29:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:29:09.795 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:29:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:29:09.795 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:29:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:29:09.795 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:29:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:29:09.795 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:29:09.795 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:29:09.795 139328 ERROR oslo_service.periodic_task >2018-06-24 19:29:39.676 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:29:39.677 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:29:39.684 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:29:39.684 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:29:39.717 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:29:39.717 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:29:39.717 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:29:39.717 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:29:39.717 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:29:39.717 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:29:39.717 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:29:39.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:29:39.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:29:39.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:29:49.721 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:29:49.755 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:29:49.755 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:29:49.755 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:29:49.755 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:29:49.755 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:29:49.755 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:29:49.755 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:29:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:29:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:29:49.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:30:09.758 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:30:09.795 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:30:09.795 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:30:09.795 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:30:09.795 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:30:09.795 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:30:09.795 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:30:09.795 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:30:09.797 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:30:09.798 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:30:09.798 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:30:09.798 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:30:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:30:09.798 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:30:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:30:09.798 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:30:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:30:09.798 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:30:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:30:09.798 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:30:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:30:09.798 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:30:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:30:09.798 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:30:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:30:09.798 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:30:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:30:09.798 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:30:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:30:09.798 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:30:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:30:09.798 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:30:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:30:09.798 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:30:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:30:09.798 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:30:09.798 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:30:09.798 139328 ERROR oslo_service.periodic_task >2018-06-24 19:30:39.680 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:30:39.681 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:30:39.688 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:30:39.689 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:30:39.727 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:30:39.727 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:30:39.727 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:30:39.727 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:30:39.727 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:30:39.727 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:30:39.727 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:30:39.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:30:39.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.041 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:30:39.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:30:49.731 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:30:49.766 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:30:49.766 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:30:49.766 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:30:49.766 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:30:49.766 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:30:49.766 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:30:49.766 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:30:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:30:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:30:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:31:09.769 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:31:09.802 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:31:09.802 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:31:09.802 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:31:09.802 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:31:09.802 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:31:09.802 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:31:09.802 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:31:09.804 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:31:09.804 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:31:09.804 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:31:09.804 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:31:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:31:09.804 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:31:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:31:09.804 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:31:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:31:09.804 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:31:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:31:09.804 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:31:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:31:09.804 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:31:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:31:09.804 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:31:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:31:09.804 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:31:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:31:09.804 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:31:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:31:09.804 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:31:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:31:09.804 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:31:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:31:09.804 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:31:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:31:09.804 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:31:09.804 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:31:09.804 139328 ERROR oslo_service.periodic_task >2018-06-24 19:31:39.676 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:31:39.677 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:32:39.674 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:32:39.675 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:32:39.681 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:32:39.682 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:32:39.716 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:32:39.716 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:32:39.716 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:32:39.716 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:32:39.716 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:32:39.716 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:32:39.716 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:32:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:32:39.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:32:39.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:32:49.720 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:32:49.752 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:32:49.752 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:32:49.752 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:32:49.752 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:32:49.752 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:32:49.752 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:32:49.752 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:32:49.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:32:49.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:32:49.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:33:09.755 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:33:09.786 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:33:09.786 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:33:09.786 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:33:09.786 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:33:09.786 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:33:09.786 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:33:09.786 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:33:09.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:33:09.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.107 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:33:09.789 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:33:09.789 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:33:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:33:09.789 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:33:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:33:09.789 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:33:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:33:09.789 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:33:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:33:09.789 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:33:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:33:09.789 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:33:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:33:09.789 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:33:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:33:09.789 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:33:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:33:09.789 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:33:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:33:09.789 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:33:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:33:09.789 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:33:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:33:09.789 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:33:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:33:09.789 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:33:09.789 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:33:09.789 139328 ERROR oslo_service.periodic_task >2018-06-24 19:33:39.670 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:33:39.671 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:33:39.680 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:33:39.681 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:33:39.722 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:33:39.722 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:33:39.722 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:33:39.722 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:33:39.722 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:33:39.722 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:33:39.722 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:33:39.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:33:39.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.045 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:33:39.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:33:49.727 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:33:49.759 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:33:49.759 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:33:49.759 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:33:49.759 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:33:49.759 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:33:49.759 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:33:49.759 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:33:49.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:33:49.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.081 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:33:49.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:34:09.762 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:34:09.794 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:34:09.794 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:34:09.794 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:34:09.794 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:34:09.794 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:34:09.794 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:34:09.794 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:34:09.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:34:09.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:34:09.796 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:34:09.796 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:34:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:34:09.796 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:34:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:34:09.796 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:34:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:34:09.796 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:34:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:34:09.796 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:34:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:34:09.796 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:34:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:34:09.796 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:34:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:34:09.796 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:34:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:34:09.796 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:34:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:34:09.796 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:34:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:34:09.796 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:34:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:34:09.796 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:34:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:34:09.796 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:34:09.796 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:34:09.796 139328 ERROR oslo_service.periodic_task >2018-06-24 19:34:39.668 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:34:39.669 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:34:39.676 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:34:39.677 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:34:39.710 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:34:39.710 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:34:39.710 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:34:39.710 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:34:39.710 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:34:39.710 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:34:39.710 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:34:39.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:34:39.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:34:39.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:34:49.713 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:34:49.746 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:34:49.746 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:34:49.746 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:34:49.746 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:34:49.746 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:34:49.746 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:34:49.746 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:34:49.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:34:49.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:34:49.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:35:09.749 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:35:09.781 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:35:09.781 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:35:09.781 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:35:09.781 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:35:09.781 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:35:09.781 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:35:09.781 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:35:09.783 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:35:09.783 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.106 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:35:09.783 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:35:09.783 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:35:09.783 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:35:09.783 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:35:09.783 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:35:09.783 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:35:09.783 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:35:09.783 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:35:09.783 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:35:09.783 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:35:09.783 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:35:09.783 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:35:09.783 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:35:09.783 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:35:09.783 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:35:09.783 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:35:09.783 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:35:09.783 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:35:09.783 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:35:09.783 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:35:09.783 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:35:09.783 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:35:09.783 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:35:09.783 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:35:09.783 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:35:09.783 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:35:09.783 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:35:09.783 139328 ERROR oslo_service.periodic_task >2018-06-24 19:35:39.665 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:35:39.666 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:35:39.673 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:35:39.673 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:35:39.706 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:35:39.706 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:35:39.706 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:35:39.706 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:35:39.706 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:35:39.706 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:35:39.706 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:35:39.709 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:35:39.709 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:35:39.709 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:35:49.710 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:35:49.745 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:35:49.745 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:35:49.745 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:35:49.745 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:35:49.745 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:35:49.745 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:35:49.745 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:35:49.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:35:49.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:35:49.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:36:09.748 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:36:09.780 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:36:09.780 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:36:09.780 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:36:09.780 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:36:09.780 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:36:09.780 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:36:09.780 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:36:09.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:36:09.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:36:09.783 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:36:09.783 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:36:09.783 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:36:09.783 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:36:09.783 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:36:09.783 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:36:09.783 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:36:09.783 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:36:09.783 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:36:09.783 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:36:09.783 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:36:09.783 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:36:09.783 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:36:09.783 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:36:09.783 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:36:09.783 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:36:09.783 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:36:09.783 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:36:09.783 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:36:09.783 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:36:09.783 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:36:09.783 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:36:09.783 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:36:09.783 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:36:09.783 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:36:09.783 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:36:09.783 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:36:09.783 139328 ERROR oslo_service.periodic_task >2018-06-24 19:36:39.664 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:36:39.665 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:36:39.673 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:36:39.673 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:36:39.709 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:36:39.709 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:36:39.709 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:36:39.709 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:36:39.709 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:36:39.709 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:36:39.709 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:36:39.711 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:36:39.711 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:36:39.711 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:36:49.713 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:36:49.749 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:36:49.749 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:36:49.749 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:36:49.749 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:36:49.749 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:36:49.749 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:36:49.749 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:36:49.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:36:49.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:36:49.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:37:09.753 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:37:09.804 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:37:09.804 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:37:09.804 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:37:09.804 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:37:09.804 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:37:09.804 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:37:09.804 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:37:09.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:37:09.807 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.133 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:37:09.807 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:37:09.807 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:37:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:37:09.807 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:37:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:37:09.807 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:37:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:37:09.807 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:37:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:37:09.807 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:37:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:37:09.807 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:37:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:37:09.807 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:37:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:37:09.807 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:37:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:37:09.807 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:37:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:37:09.807 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:37:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:37:09.807 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:37:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:37:09.807 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:37:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:37:09.807 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:37:09.807 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:37:09.807 139328 ERROR oslo_service.periodic_task >2018-06-24 19:37:39.668 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:37:39.669 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:37:39.676 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:37:39.677 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:37:39.710 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:37:39.710 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:37:39.710 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:37:39.710 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:37:39.710 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:37:39.710 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:37:39.710 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:37:39.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:37:39.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:37:39.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:37:49.713 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:37:49.749 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:37:49.749 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:37:49.749 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:37:49.749 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:37:49.749 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:37:49.749 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:37:49.749 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:37:49.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:37:49.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:37:49.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:38:09.752 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:38:09.785 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:38:09.785 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:38:09.785 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:38:09.785 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:38:09.785 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:38:09.785 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:38:09.785 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:38:09.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:38:09.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:38:09.788 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:38:09.788 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:38:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:38:09.788 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:38:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:38:09.788 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:38:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:38:09.788 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:38:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:38:09.788 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:38:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:38:09.788 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:38:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:38:09.788 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:38:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:38:09.788 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:38:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:38:09.788 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:38:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:38:09.788 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:38:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:38:09.788 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:38:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:38:09.788 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:38:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:38:09.788 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:38:09.788 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:38:09.788 139328 ERROR oslo_service.periodic_task >2018-06-24 19:38:39.669 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:38:39.670 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:38:39.676 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:38:39.676 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:38:39.709 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:38:39.709 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:38:39.709 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:38:39.709 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:38:39.709 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:38:39.709 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:38:39.709 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:38:39.711 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:38:39.711 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:38:39.711 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:38:49.712 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:38:49.744 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:38:49.744 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:38:49.744 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:38:49.744 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:38:49.744 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:38:49.744 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:38:49.744 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:38:49.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:38:49.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.07 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:38:49.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:39:09.747 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:39:09.782 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:39:09.782 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:39:09.782 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:39:09.782 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:39:09.782 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:39:09.782 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:39:09.782 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:39:09.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:39:09.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.107 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:39:09.784 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:39:09.784 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:39:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:39:09.784 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:39:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:39:09.784 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:39:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:39:09.784 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:39:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:39:09.784 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:39:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:39:09.784 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:39:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:39:09.784 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:39:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:39:09.784 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:39:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:39:09.784 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:39:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:39:09.784 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:39:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:39:09.784 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:39:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:39:09.784 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:39:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:39:09.784 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:39:09.784 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:39:09.784 139328 ERROR oslo_service.periodic_task >2018-06-24 19:39:39.666 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:39:39.667 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:39:39.677 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:39:39.678 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:39:39.720 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:39:39.720 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:39:39.720 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:39:39.720 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:39:39.720 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:39:39.720 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:39:39.720 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:39:39.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:39:39.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.044 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:39:39.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:39:49.723 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:39:49.755 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:39:49.755 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:39:49.755 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:39:49.755 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:39:49.755 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:39:49.755 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:39:49.755 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:39:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:39:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:39:49.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:40:09.758 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:40:09.791 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:40:09.791 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:40:09.791 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:40:09.791 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:40:09.791 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:40:09.791 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:40:09.791 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:40:09.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:40:09.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:40:09.793 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:40:09.793 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:40:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:40:09.793 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:40:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:40:09.793 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:40:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:40:09.793 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:40:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:40:09.793 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:40:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:40:09.793 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:40:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:40:09.793 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:40:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:40:09.793 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:40:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:40:09.793 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:40:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:40:09.793 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:40:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:40:09.793 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:40:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:40:09.793 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:40:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:40:09.793 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:40:09.793 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:40:09.793 139328 ERROR oslo_service.periodic_task >2018-06-24 19:40:39.665 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:40:39.666 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:40:39.675 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:40:39.676 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:40:39.712 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:40:39.712 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:40:39.712 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:40:39.712 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:40:39.712 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:40:39.712 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:40:39.712 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:40:39.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:40:39.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:40:39.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:40:49.715 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:40:49.747 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:40:49.747 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:40:49.747 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:40:49.747 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:40:49.747 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:40:49.747 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:40:49.747 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:40:49.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:40:49.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:40:49.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:41:09.750 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:41:09.789 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:41:09.789 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:41:09.789 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:41:09.789 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:41:09.789 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:41:09.789 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:41:09.789 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:41:09.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:41:09.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.118 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:41:09.794 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:41:09.794 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:41:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:41:09.794 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:41:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:41:09.794 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:41:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:41:09.794 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:41:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:41:09.794 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:41:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:41:09.794 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:41:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:41:09.794 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:41:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:41:09.794 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:41:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:41:09.794 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:41:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:41:09.794 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:41:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:41:09.794 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:41:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:41:09.794 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:41:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:41:09.794 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:41:09.794 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:41:09.794 139328 ERROR oslo_service.periodic_task >2018-06-24 19:41:39.666 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:41:39.667 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:41:39.674 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:41:39.675 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:41:39.712 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:41:39.712 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:41:39.712 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:41:39.712 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:41:39.712 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:41:39.712 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:41:39.712 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:41:39.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:41:39.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:41:39.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:41:49.716 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:41:49.747 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:41:49.747 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:41:49.747 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:41:49.747 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:41:49.747 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:41:49.747 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:41:49.747 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:41:49.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:41:49.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:41:49.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:42:09.750 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:42:09.786 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:42:09.786 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:42:09.786 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:42:09.786 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:42:09.786 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:42:09.786 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:42:09.786 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:42:09.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:42:09.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:42:09.788 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:42:09.788 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:42:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:42:09.788 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:42:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:42:09.788 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:42:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:42:09.788 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:42:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:42:09.788 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:42:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:42:09.788 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:42:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:42:09.788 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:42:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:42:09.788 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:42:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:42:09.788 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:42:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:42:09.788 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:42:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:42:09.788 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:42:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:42:09.788 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:42:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:42:09.788 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:42:09.788 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:42:09.788 139328 ERROR oslo_service.periodic_task >2018-06-24 19:42:39.670 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:42:39.671 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:42:39.677 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:42:39.678 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:42:39.713 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:42:39.713 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:42:39.713 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:42:39.713 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:42:39.713 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:42:39.713 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:42:39.713 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:42:39.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:42:39.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:42:39.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:42:49.716 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:42:49.749 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:42:49.749 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:42:49.749 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:42:49.749 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:42:49.749 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:42:49.749 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:42:49.749 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:42:49.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:42:49.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:42:49.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:43:09.751 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:43:09.784 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:43:09.784 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:43:09.784 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:43:09.784 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:43:09.784 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:43:09.784 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:43:09.784 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:43:09.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:43:09.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:43:09.788 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:43:09.788 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:43:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:43:09.788 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:43:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:43:09.788 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:43:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:43:09.788 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:43:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:43:09.788 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:43:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:43:09.788 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:43:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:43:09.788 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:43:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:43:09.788 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:43:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:43:09.788 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:43:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:43:09.788 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:43:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:43:09.788 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:43:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:43:09.788 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:43:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:43:09.788 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:43:09.788 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:43:09.788 139328 ERROR oslo_service.periodic_task >2018-06-24 19:43:39.670 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:43:39.671 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:43:39.679 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:43:39.680 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:43:39.716 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:43:39.716 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:43:39.716 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:43:39.716 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:43:39.716 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:43:39.716 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:43:39.716 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:43:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:43:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:43:39.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:43:49.719 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:43:49.751 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:43:49.751 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:43:49.751 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:43:49.751 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:43:49.751 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:43:49.751 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:43:49.751 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:43:49.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:43:49.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:43:49.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:44:09.755 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:44:09.791 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:44:09.791 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:44:09.791 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:44:09.791 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:44:09.791 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:44:09.791 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:44:09.791 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:44:09.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:44:09.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:44:09.794 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:44:09.794 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:44:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:44:09.794 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:44:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:44:09.794 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:44:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:44:09.794 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:44:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:44:09.794 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:44:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:44:09.794 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:44:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:44:09.794 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:44:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:44:09.794 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:44:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:44:09.794 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:44:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:44:09.794 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:44:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:44:09.794 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:44:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:44:09.794 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:44:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:44:09.794 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:44:09.794 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:44:09.794 139328 ERROR oslo_service.periodic_task >2018-06-24 19:44:39.665 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:44:39.666 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:44:39.673 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:44:39.674 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:44:39.708 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:44:39.708 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:44:39.708 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:44:39.708 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:44:39.708 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:44:39.708 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:44:39.708 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:44:39.710 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:44:39.710 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:44:39.711 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:44:49.711 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:44:49.747 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:44:49.747 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:44:49.747 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:44:49.747 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:44:49.747 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:44:49.747 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:44:49.747 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:44:49.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:44:49.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:44:49.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:45:09.751 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:45:09.786 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:45:09.786 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:45:09.786 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:45:09.786 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:45:09.786 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:45:09.786 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:45:09.786 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:45:09.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:45:09.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:45:09.789 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:45:09.789 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:45:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:45:09.789 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:45:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:45:09.789 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:45:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:45:09.789 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:45:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:45:09.789 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:45:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:45:09.789 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:45:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:45:09.789 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:45:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:45:09.789 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:45:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:45:09.789 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:45:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:45:09.789 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:45:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:45:09.789 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:45:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:45:09.789 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:45:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:45:09.789 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:45:09.789 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:45:09.789 139328 ERROR oslo_service.periodic_task >2018-06-24 19:45:39.660 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:45:39.662 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:45:39.673 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:45:39.673 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:45:39.706 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:45:39.706 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:45:39.706 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:45:39.706 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:45:39.706 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:45:39.706 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:45:39.706 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:45:39.708 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:45:39.708 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:45:39.709 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:45:49.709 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:45:49.742 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:45:49.742 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:45:49.742 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:45:49.742 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:45:49.742 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:45:49.742 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:45:49.742 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:45:49.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:45:49.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:45:49.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:46:09.747 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:46:09.779 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:46:09.779 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:46:09.779 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:46:09.779 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:46:09.779 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:46:09.779 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:46:09.779 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:46:09.781 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:46:09.781 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.107 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:46:09.781 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:46:09.781 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:46:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:46:09.781 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:46:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:46:09.781 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:46:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:46:09.781 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:46:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:46:09.781 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:46:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:46:09.781 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:46:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:46:09.781 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:46:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:46:09.781 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:46:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:46:09.781 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:46:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:46:09.781 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:46:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:46:09.781 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:46:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:46:09.781 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:46:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:46:09.781 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:46:09.781 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:46:09.781 139328 ERROR oslo_service.periodic_task >2018-06-24 19:46:39.663 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:46:39.664 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:46:39.670 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:46:39.670 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:46:39.703 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:46:39.703 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:46:39.703 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:46:39.703 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:46:39.703 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:46:39.703 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:46:39.703 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:46:39.706 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:46:39.706 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:46:39.706 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:46:49.707 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:46:49.739 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:46:49.739 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:46:49.739 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:46:49.739 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:46:49.739 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:46:49.739 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:46:49.739 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:46:49.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:46:49.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:46:49.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:47:09.742 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:47:09.774 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:47:09.774 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:47:09.774 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:47:09.774 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:47:09.774 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:47:09.774 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:47:09.774 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:47:09.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:47:09.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.106 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:47:09.777 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:47:09.777 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:47:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:47:09.777 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:47:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:47:09.777 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:47:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:47:09.777 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:47:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:47:09.777 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:47:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:47:09.777 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:47:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:47:09.777 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:47:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:47:09.777 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:47:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:47:09.777 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:47:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:47:09.777 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:47:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:47:09.777 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:47:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:47:09.777 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:47:09.777 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:47:09.777 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:47:09.777 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:47:09.777 139328 ERROR oslo_service.periodic_task >2018-06-24 19:47:39.658 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:47:39.659 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:47:39.666 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:47:39.667 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:47:39.698 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:47:39.698 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:47:39.698 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:47:39.698 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:47:39.698 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:47:39.698 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:47:39.698 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:47:39.700 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:47:39.701 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:47:39.701 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:47:49.701 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:47:49.738 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:47:49.738 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:47:49.738 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:47:49.738 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:47:49.738 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:47:49.738 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:47:49.738 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:47:49.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:47:49.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:47:49.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:48:09.741 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:48:09.777 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:48:09.777 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:48:09.777 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:48:09.777 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:48:09.777 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:48:09.777 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:48:09.777 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:48:09.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:48:09.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:48:09.781 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:48:09.781 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:48:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:48:09.781 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:48:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:48:09.781 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:48:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:48:09.781 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:48:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:48:09.781 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:48:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:48:09.781 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:48:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:48:09.781 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:48:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:48:09.781 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:48:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:48:09.781 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:48:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:48:09.781 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:48:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:48:09.781 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:48:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:48:09.781 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:48:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:48:09.781 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:48:09.781 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:48:09.781 139328 ERROR oslo_service.periodic_task >2018-06-24 19:48:39.662 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:48:39.663 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:48:39.672 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:48:39.673 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:48:39.709 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:48:39.709 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:48:39.709 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:48:39.709 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:48:39.709 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:48:39.709 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:48:39.709 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:48:39.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:48:39.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:48:39.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:48:49.713 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:48:49.745 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:48:49.745 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:48:49.745 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:48:49.745 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:48:49.745 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:48:49.745 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:48:49.745 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:48:49.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:48:49.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:48:49.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:49:09.748 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:49:09.780 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:49:09.780 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:49:09.780 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:49:09.780 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:49:09.780 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:49:09.780 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:49:09.780 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:49:09.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:49:09.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:49:09.785 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:49:09.785 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:49:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:49:09.785 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:49:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:49:09.785 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:49:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:49:09.785 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:49:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:49:09.785 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:49:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:49:09.785 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:49:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:49:09.785 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:49:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:49:09.785 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:49:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:49:09.785 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:49:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:49:09.785 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:49:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:49:09.785 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:49:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:49:09.785 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:49:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:49:09.785 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:49:09.785 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:49:09.785 139328 ERROR oslo_service.periodic_task >2018-06-24 19:49:39.667 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:49:39.667 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:49:39.675 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:49:39.676 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:49:39.709 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:49:39.709 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:49:39.709 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:49:39.709 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:49:39.709 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:49:39.709 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:49:39.709 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:49:39.711 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:49:39.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:49:39.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:49:49.713 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:49:49.745 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:49:49.745 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:49:49.745 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:49:49.745 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:49:49.745 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:49:49.745 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:49:49.745 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:49:49.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:49:49.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:49:49.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:50:09.748 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:50:09.781 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:50:09.781 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:50:09.781 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:50:09.781 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:50:09.781 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:50:09.781 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:50:09.781 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:50:09.783 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:50:09.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:50:09.784 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:50:09.784 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:50:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:50:09.784 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:50:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:50:09.784 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:50:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:50:09.784 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:50:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:50:09.784 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:50:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:50:09.784 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:50:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:50:09.784 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:50:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:50:09.784 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:50:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:50:09.784 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:50:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:50:09.784 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:50:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:50:09.784 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:50:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:50:09.784 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:50:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:50:09.784 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:50:09.784 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:50:09.784 139328 ERROR oslo_service.periodic_task >2018-06-24 19:50:39.666 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:50:39.667 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:50:39.674 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:50:39.675 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:50:39.711 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:50:39.711 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:50:39.711 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:50:39.711 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:50:39.711 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:50:39.711 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:50:39.711 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:50:39.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:50:39.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:50:39.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:50:49.715 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:50:49.752 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:50:49.752 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:50:49.752 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:50:49.752 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:50:49.752 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:50:49.752 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:50:49.752 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:50:49.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:50:49.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:50:49.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:51:09.755 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:51:09.786 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:51:09.786 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:51:09.786 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:51:09.786 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:51:09.786 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:51:09.786 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:51:09.786 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:51:09.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:51:09.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:51:09.789 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:51:09.789 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:51:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:51:09.789 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:51:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:51:09.789 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:51:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:51:09.789 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:51:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:51:09.789 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:51:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:51:09.789 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:51:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:51:09.789 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:51:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:51:09.789 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:51:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:51:09.789 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:51:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:51:09.789 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:51:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:51:09.789 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:51:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:51:09.789 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:51:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:51:09.789 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:51:09.789 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:51:09.789 139328 ERROR oslo_service.periodic_task >2018-06-24 19:51:39.671 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:51:39.671 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:51:39.679 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:51:39.679 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:51:39.718 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:51:39.718 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:51:39.718 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:51:39.718 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:51:39.718 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:51:39.718 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:51:39.718 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:51:39.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:51:39.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.041 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:51:39.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:51:49.722 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:51:49.754 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:51:49.754 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:51:49.754 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:51:49.754 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:51:49.754 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:51:49.754 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:51:49.754 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:51:49.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:51:49.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:51:49.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:52:09.757 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:52:09.796 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:52:09.796 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:52:09.796 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:52:09.796 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:52:09.796 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:52:09.796 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:52:09.796 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:52:09.798 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:52:09.799 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.119 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:52:09.799 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:52:09.799 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:52:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:52:09.799 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:52:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:52:09.799 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:52:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:52:09.799 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:52:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:52:09.799 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:52:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:52:09.799 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:52:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:52:09.799 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:52:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:52:09.799 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:52:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:52:09.799 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:52:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:52:09.799 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:52:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:52:09.799 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:52:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:52:09.799 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:52:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:52:09.799 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:52:09.799 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:52:09.799 139328 ERROR oslo_service.periodic_task >2018-06-24 19:52:39.670 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:52:39.671 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:52:39.678 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:52:39.679 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:52:39.712 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:52:39.712 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:52:39.712 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:52:39.712 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:52:39.712 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:52:39.712 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:52:39.712 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:52:39.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:52:39.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:52:39.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:52:49.715 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:52:49.748 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:52:49.748 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:52:49.748 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:52:49.748 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:52:49.748 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:52:49.748 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:52:49.748 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:52:49.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:52:49.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:52:49.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:53:09.751 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:53:09.784 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:53:09.784 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:53:09.784 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:53:09.784 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:53:09.784 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:53:09.784 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:53:09.784 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:53:09.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:53:09.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:53:09.787 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:53:09.787 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:53:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:53:09.787 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:53:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:53:09.787 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:53:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:53:09.787 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:53:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:53:09.787 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:53:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:53:09.787 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:53:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:53:09.787 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:53:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:53:09.787 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:53:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:53:09.787 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:53:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:53:09.787 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:53:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:53:09.787 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:53:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:53:09.787 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:53:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:53:09.787 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:53:09.787 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:53:09.787 139328 ERROR oslo_service.periodic_task >2018-06-24 19:53:39.668 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:53:39.669 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:53:39.675 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:53:39.676 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:53:39.710 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:53:39.710 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:53:39.710 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:53:39.710 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:53:39.710 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:53:39.710 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:53:39.710 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:53:39.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:53:39.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:53:39.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:53:49.713 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:53:49.747 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:53:49.747 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:53:49.747 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:53:49.747 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:53:49.747 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:53:49.747 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:53:49.747 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:53:49.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:53:49.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:53:49.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:54:09.750 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:54:09.790 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:54:09.790 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:54:09.790 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:54:09.790 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:54:09.790 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:54:09.790 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:54:09.790 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:54:09.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:54:09.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:54:09.793 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:54:09.793 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:54:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:54:09.793 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:54:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:54:09.793 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:54:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:54:09.793 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:54:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:54:09.793 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:54:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:54:09.793 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:54:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:54:09.793 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:54:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:54:09.793 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:54:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:54:09.793 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:54:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:54:09.793 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:54:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:54:09.793 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:54:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:54:09.793 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:54:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:54:09.793 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:54:09.793 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:54:09.793 139328 ERROR oslo_service.periodic_task >2018-06-24 19:54:39.664 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:54:39.665 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:54:39.674 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:54:39.674 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:54:39.710 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:54:39.710 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:54:39.710 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:54:39.710 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:54:39.710 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:54:39.710 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:54:39.710 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:54:39.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:54:39.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:54:39.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:54:49.714 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:54:49.746 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:54:49.746 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:54:49.746 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:54:49.746 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:54:49.746 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:54:49.746 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:54:49.746 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:54:49.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:54:49.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:54:49.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:55:09.749 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:55:09.783 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:55:09.783 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:55:09.783 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:55:09.783 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:55:09.783 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:55:09.783 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:55:09.783 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:55:09.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:55:09.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:55:09.786 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:55:09.786 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:55:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:55:09.786 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:55:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:55:09.786 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:55:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:55:09.786 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:55:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:55:09.786 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:55:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:55:09.786 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:55:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:55:09.786 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:55:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:55:09.786 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:55:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:55:09.786 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:55:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:55:09.786 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:55:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:55:09.786 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:55:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:55:09.786 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:55:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:55:09.786 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:55:09.786 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:55:09.786 139328 ERROR oslo_service.periodic_task >2018-06-24 19:55:39.667 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:55:39.668 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:55:39.675 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:55:39.676 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:55:39.712 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:55:39.712 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:55:39.712 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:55:39.712 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:55:39.712 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:55:39.712 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:55:39.712 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:55:39.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:55:39.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:55:39.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:55:49.715 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:55:49.748 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:55:49.748 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:55:49.748 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:55:49.748 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:55:49.748 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:55:49.748 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:55:49.748 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:55:49.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:55:49.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:55:49.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:56:09.752 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:56:09.785 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:56:09.785 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:56:09.785 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:56:09.785 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:56:09.785 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:56:09.785 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:56:09.785 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:56:09.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:56:09.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:56:09.788 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:56:09.788 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:56:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:56:09.788 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:56:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:56:09.788 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:56:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:56:09.788 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:56:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:56:09.788 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:56:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:56:09.788 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:56:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:56:09.788 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:56:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:56:09.788 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:56:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:56:09.788 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:56:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:56:09.788 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:56:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:56:09.788 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:56:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:56:09.788 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:56:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:56:09.788 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:56:09.788 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:56:09.788 139328 ERROR oslo_service.periodic_task >2018-06-24 19:56:39.669 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:56:39.671 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:56:39.678 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:56:39.678 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:56:39.712 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:56:39.712 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:56:39.712 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:56:39.712 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:56:39.712 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:56:39.712 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:56:39.712 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:56:39.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:56:39.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:56:39.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:56:49.715 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:56:49.747 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:56:49.747 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:56:49.747 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:56:49.747 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:56:49.747 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:56:49.747 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:56:49.747 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:56:49.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:56:49.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:56:49.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:57:09.750 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:57:09.789 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:57:09.789 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:57:09.789 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:57:09.789 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:57:09.789 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:57:09.789 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:57:09.789 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:57:09.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:57:09.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:57:09.793 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:57:09.793 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:57:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:57:09.793 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:57:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:57:09.793 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:57:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:57:09.793 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:57:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:57:09.793 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:57:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:57:09.793 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:57:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:57:09.793 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:57:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:57:09.793 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:57:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:57:09.793 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:57:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:57:09.793 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:57:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:57:09.793 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:57:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:57:09.793 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:57:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:57:09.793 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:57:09.793 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:57:09.793 139328 ERROR oslo_service.periodic_task >2018-06-24 19:57:39.675 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:57:39.676 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:57:39.683 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:57:39.684 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:57:39.734 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:57:39.734 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:57:39.734 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:57:39.734 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:57:39.734 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:57:39.734 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:57:39.734 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:57:39.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:57:39.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.053 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:57:39.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:57:49.738 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:57:49.770 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:57:49.770 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:57:49.770 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:57:49.770 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:57:49.770 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:57:49.770 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:57:49.770 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:57:49.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:57:49.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.088 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:57:49.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:58:09.773 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:58:09.806 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:58:09.806 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:58:09.806 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:58:09.806 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:58:09.806 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:58:09.806 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:58:09.806 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:58:09.808 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:58:09.809 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.125 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:58:09.809 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:58:09.809 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:58:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:58:09.809 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:58:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:58:09.809 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:58:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:58:09.809 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:58:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:58:09.809 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:58:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:58:09.809 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:58:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:58:09.809 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:58:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:58:09.809 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:58:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:58:09.809 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:58:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:58:09.809 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:58:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:58:09.809 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:58:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:58:09.809 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:58:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:58:09.809 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:58:09.809 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:58:09.809 139328 ERROR oslo_service.periodic_task >2018-06-24 19:58:39.670 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:58:39.671 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:58:39.703 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:58:39.703 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:58:39.703 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:58:39.703 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:58:39.703 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:58:39.703 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:58:39.703 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:58:39.705 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:58:39.705 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:58:39.706 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:58:49.706 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:58:49.739 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:58:49.739 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:58:49.739 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:58:49.739 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:58:49.739 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:58:49.739 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:58:49.739 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:58:49.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:58:49.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.07 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:58:49.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:59:09.742 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:59:09.775 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:59:09.775 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:59:09.775 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:59:09.775 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:59:09.775 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:59:09.775 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:59:09.775 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:59:09.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:59:09.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.106 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:59:09.778 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:59:09.778 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 19:59:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 19:59:09.778 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 19:59:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 19:59:09.778 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 19:59:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 19:59:09.778 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 19:59:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 19:59:09.778 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 19:59:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 19:59:09.778 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 19:59:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 19:59:09.778 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 19:59:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 19:59:09.778 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 19:59:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 19:59:09.778 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 19:59:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 19:59:09.778 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 19:59:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 19:59:09.778 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 19:59:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 19:59:09.778 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 19:59:09.778 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 19:59:09.778 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 19:59:09.778 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 19:59:09.778 139328 ERROR oslo_service.periodic_task >2018-06-24 19:59:39.670 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:59:39.671 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 19:59:39.679 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 19:59:39.680 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:59:39.715 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:59:39.715 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:59:39.715 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:59:39.715 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:59:39.715 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:59:39.715 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:59:39.715 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:59:39.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:59:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:59:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 19:59:49.718 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 19:59:49.751 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 19:59:49.751 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 19:59:49.751 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 19:59:49.751 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 19:59:49.751 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 19:59:49.751 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 19:59:49.751 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 19:59:49.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 19:59:49.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 19:59:49.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:00:09.754 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:00:09.788 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:00:09.788 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:00:09.788 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:00:09.788 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:00:09.788 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:00:09.788 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:00:09.788 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:00:09.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:00:09.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:00:09.790 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:00:09.790 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:00:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:00:09.790 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:00:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:00:09.790 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:00:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:00:09.790 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:00:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:00:09.790 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:00:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:00:09.790 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:00:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:00:09.790 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:00:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:00:09.790 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:00:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:00:09.790 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:00:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:00:09.790 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:00:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:00:09.790 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:00:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:00:09.790 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:00:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:00:09.790 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:00:09.790 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:00:09.790 139328 ERROR oslo_service.periodic_task >2018-06-24 20:00:39.672 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:00:39.673 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:00:39.681 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:00:39.681 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:00:39.716 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:00:39.716 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:00:39.716 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:00:39.716 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:00:39.716 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:00:39.716 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:00:39.716 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:00:39.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:00:39.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:00:39.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:00:49.720 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:00:49.755 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:00:49.755 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:00:49.755 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:00:49.755 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:00:49.755 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:00:49.755 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:00:49.755 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:00:49.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:00:49.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:00:49.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:01:09.759 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:01:09.793 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:01:09.793 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:01:09.793 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:01:09.793 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:01:09.793 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:01:09.793 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:01:09.793 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:01:09.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:01:09.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:01:09.796 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:01:09.796 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:01:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:01:09.796 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:01:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:01:09.796 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:01:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:01:09.796 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:01:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:01:09.796 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:01:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:01:09.796 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:01:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:01:09.796 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:01:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:01:09.796 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:01:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:01:09.796 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:01:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:01:09.796 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:01:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:01:09.796 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:01:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:01:09.796 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:01:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:01:09.796 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:01:09.796 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:01:09.796 139328 ERROR oslo_service.periodic_task >2018-06-24 20:01:39.668 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:01:39.669 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:01:39.676 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:01:39.676 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:01:39.711 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:01:39.711 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:01:39.711 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:01:39.711 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:01:39.711 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:01:39.711 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:01:39.711 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:01:39.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:01:39.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:01:39.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:01:49.715 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:01:49.752 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:01:49.752 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:01:49.752 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:01:49.752 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:01:49.752 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:01:49.752 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:01:49.752 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:01:49.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:01:49.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:01:49.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:02:09.755 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:02:09.789 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:02:09.789 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:02:09.789 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:02:09.789 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:02:09.789 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:02:09.789 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:02:09.789 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:02:09.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:02:09.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:02:09.791 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:02:09.791 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:02:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:02:09.791 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:02:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:02:09.791 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:02:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:02:09.791 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:02:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:02:09.791 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:02:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:02:09.791 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:02:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:02:09.791 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:02:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:02:09.791 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:02:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:02:09.791 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:02:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:02:09.791 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:02:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:02:09.791 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:02:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:02:09.791 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:02:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:02:09.791 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:02:09.791 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:02:09.791 139328 ERROR oslo_service.periodic_task >2018-06-24 20:02:39.672 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:02:39.674 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:02:39.682 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:02:39.683 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:02:39.717 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:02:39.717 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:02:39.717 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:02:39.717 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:02:39.717 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:02:39.717 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:02:39.717 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:02:39.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:02:39.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:02:39.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:02:49.721 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:02:49.755 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:02:49.755 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:02:49.755 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:02:49.755 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:02:49.755 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:02:49.755 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:02:49.755 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:02:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:02:49.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:02:49.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:03:09.759 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:03:09.792 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:03:09.792 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:03:09.792 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:03:09.792 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:03:09.792 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:03:09.792 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:03:09.792 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:03:09.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:03:09.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:03:09.795 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:03:09.795 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:03:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:03:09.795 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:03:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:03:09.795 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:03:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:03:09.795 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:03:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:03:09.795 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:03:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:03:09.795 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:03:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:03:09.795 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:03:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:03:09.795 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:03:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:03:09.795 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:03:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:03:09.795 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:03:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:03:09.795 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:03:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:03:09.795 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:03:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:03:09.795 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:03:09.795 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:03:09.795 139328 ERROR oslo_service.periodic_task >2018-06-24 20:03:39.676 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:03:39.677 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:03:39.686 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:03:39.687 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:03:39.722 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:03:39.722 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:03:39.722 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:03:39.722 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:03:39.722 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:03:39.722 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:03:39.722 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:03:39.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:03:39.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:03:39.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:03:49.727 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:03:49.759 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:03:49.759 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:03:49.759 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:03:49.759 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:03:49.759 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:03:49.759 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:03:49.759 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:03:49.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:03:49.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:03:49.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:04:09.762 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:04:09.807 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:04:09.807 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:04:09.807 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:04:09.807 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:04:09.807 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:04:09.807 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:04:09.807 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:04:09.810 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:04:09.810 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.123 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:04:09.811 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:04:09.811 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:04:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:04:09.811 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:04:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:04:09.811 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:04:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:04:09.811 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:04:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:04:09.811 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:04:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:04:09.811 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:04:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:04:09.811 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:04:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:04:09.811 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:04:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:04:09.811 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:04:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:04:09.811 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:04:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:04:09.811 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:04:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:04:09.811 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:04:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:04:09.811 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:04:09.811 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:04:09.811 139328 ERROR oslo_service.periodic_task >2018-06-24 20:04:39.672 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:04:39.673 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:04:39.680 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:04:39.681 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:04:39.721 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:04:39.721 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:04:39.721 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:04:39.721 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:04:39.721 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:04:39.721 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:04:39.721 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:04:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:04:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.043 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:04:39.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:04:49.725 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:04:49.761 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:04:49.761 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:04:49.761 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:04:49.761 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:04:49.761 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:04:49.761 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:04:49.761 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:04:49.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:04:49.764 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.083 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:04:49.764 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:05:09.764 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:05:09.797 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:05:09.797 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:05:09.797 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:05:09.797 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:05:09.797 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:05:09.797 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:05:09.797 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:05:09.799 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:05:09.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.119 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:05:09.800 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:05:09.800 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:05:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:05:09.800 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:05:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:05:09.800 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:05:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:05:09.800 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:05:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:05:09.800 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:05:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:05:09.800 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:05:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:05:09.800 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:05:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:05:09.800 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:05:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:05:09.800 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:05:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:05:09.800 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:05:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:05:09.800 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:05:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:05:09.800 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:05:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:05:09.800 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:05:09.800 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:05:09.800 139328 ERROR oslo_service.periodic_task >2018-06-24 20:05:39.671 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:05:39.672 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:05:39.681 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:05:39.681 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:05:39.716 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:05:39.716 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:05:39.716 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:05:39.716 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:05:39.716 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:05:39.716 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:05:39.716 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:05:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:05:39.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:05:39.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:05:49.720 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:05:49.755 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:05:49.755 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:05:49.755 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:05:49.755 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:05:49.755 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:05:49.755 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:05:49.755 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:05:49.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:05:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:05:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:06:09.758 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:06:09.792 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:06:09.792 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:06:09.792 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:06:09.792 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:06:09.792 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:06:09.792 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:06:09.792 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:06:09.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:06:09.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:06:09.795 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:06:09.795 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:06:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:06:09.795 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:06:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:06:09.795 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:06:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:06:09.795 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:06:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:06:09.795 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:06:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:06:09.795 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:06:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:06:09.795 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:06:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:06:09.795 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:06:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:06:09.795 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:06:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:06:09.795 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:06:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:06:09.795 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:06:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:06:09.795 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:06:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:06:09.795 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:06:09.795 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:06:09.795 139328 ERROR oslo_service.periodic_task >2018-06-24 20:06:39.667 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:06:39.668 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:06:39.680 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:06:39.681 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:06:39.714 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:06:39.714 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:06:39.714 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:06:39.714 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:06:39.714 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:06:39.714 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:06:39.714 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:06:39.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:06:39.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:06:39.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:06:49.717 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:06:49.749 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:06:49.749 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:06:49.749 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:06:49.749 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:06:49.749 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:06:49.749 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:06:49.749 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:06:49.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:06:49.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.07 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:06:49.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:07:09.753 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:07:09.786 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:07:09.786 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:07:09.786 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:07:09.786 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:07:09.786 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:07:09.786 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:07:09.786 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:07:09.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:07:09.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.107 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:07:09.789 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:07:09.789 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:07:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:07:09.789 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:07:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:07:09.789 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:07:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:07:09.789 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:07:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:07:09.789 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:07:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:07:09.789 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:07:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:07:09.789 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:07:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:07:09.789 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:07:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:07:09.789 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:07:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:07:09.789 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:07:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:07:09.789 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:07:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:07:09.789 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:07:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:07:09.789 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:07:09.789 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:07:09.789 139328 ERROR oslo_service.periodic_task >2018-06-24 20:07:39.671 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:07:39.672 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:07:39.679 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:07:39.680 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:07:39.715 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:07:39.715 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:07:39.715 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:07:39.715 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:07:39.715 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:07:39.715 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:07:39.715 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:07:39.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:07:39.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:07:39.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:07:49.718 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:07:49.754 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:07:49.754 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:07:49.754 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:07:49.754 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:07:49.754 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:07:49.754 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:07:49.754 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:07:49.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:07:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:07:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:08:09.758 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:08:09.791 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:08:09.791 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:08:09.791 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:08:09.791 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:08:09.791 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:08:09.791 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:08:09.791 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:08:09.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:08:09.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:08:09.793 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:08:09.793 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:08:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:08:09.793 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:08:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:08:09.793 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:08:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:08:09.793 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:08:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:08:09.793 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:08:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:08:09.793 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:08:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:08:09.793 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:08:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:08:09.793 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:08:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:08:09.793 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:08:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:08:09.793 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:08:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:08:09.793 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:08:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:08:09.793 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:08:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:08:09.793 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:08:09.793 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:08:09.793 139328 ERROR oslo_service.periodic_task >2018-06-24 20:08:39.675 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:08:39.676 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:08:39.682 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:08:39.683 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:08:39.715 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:08:39.715 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:08:39.715 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:08:39.715 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:08:39.715 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:08:39.715 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:08:39.715 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:08:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:08:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:08:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:08:49.719 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:08:49.753 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:08:49.753 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:08:49.753 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:08:49.753 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:08:49.753 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:08:49.753 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:08:49.753 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:08:49.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:08:49.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:08:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:09:09.757 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:09:09.790 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:09:09.790 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:09:09.790 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:09:09.790 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:09:09.790 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:09:09.790 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:09:09.790 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:09:09.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:09:09.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:09:09.793 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:09:09.793 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:09:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:09:09.793 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:09:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:09:09.793 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:09:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:09:09.793 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:09:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:09:09.793 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:09:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:09:09.793 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:09:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:09:09.793 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:09:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:09:09.793 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:09:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:09:09.793 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:09:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:09:09.793 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:09:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:09:09.793 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:09:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:09:09.793 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:09:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:09:09.793 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:09:09.793 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:09:09.793 139328 ERROR oslo_service.periodic_task >2018-06-24 20:09:39.674 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:09:39.676 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:09:39.683 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:09:39.684 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:09:39.735 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:09:39.735 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:09:39.735 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:09:39.735 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:09:39.735 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:09:39.735 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:09:39.735 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:09:39.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:09:39.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.054 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:09:39.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:09:49.738 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:09:49.771 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:09:49.771 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:09:49.771 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:09:49.771 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:09:49.771 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:09:49.771 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:09:49.771 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:09:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:09:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.089 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:09:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:10:09.774 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:10:09.809 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:10:09.809 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:10:09.809 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:10:09.809 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:10:09.809 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:10:09.809 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:10:09.809 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:10:09.811 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:10:09.811 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.127 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:10:09.811 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:10:09.811 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:10:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:10:09.811 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:10:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:10:09.811 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:10:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:10:09.811 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:10:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:10:09.811 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:10:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:10:09.811 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:10:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:10:09.811 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:10:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:10:09.811 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:10:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:10:09.811 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:10:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:10:09.811 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:10:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:10:09.811 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:10:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:10:09.811 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:10:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:10:09.811 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:10:09.811 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:10:09.811 139328 ERROR oslo_service.periodic_task >2018-06-24 20:10:39.673 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:10:39.674 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:10:39.682 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:10:39.682 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:10:39.717 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:10:39.717 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:10:39.717 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:10:39.717 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:10:39.717 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:10:39.717 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:10:39.717 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:10:39.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:10:39.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:10:39.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:10:49.721 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:10:49.758 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:10:49.758 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:10:49.758 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:10:49.758 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:10:49.758 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:10:49.758 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:10:49.758 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:10:49.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:10:49.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:10:49.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:11:09.761 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:11:09.796 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:11:09.796 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:11:09.796 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:11:09.796 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:11:09.796 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:11:09.796 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:11:09.796 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:11:09.799 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:11:09.799 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:11:09.800 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:11:09.800 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:11:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:11:09.800 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:11:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:11:09.800 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:11:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:11:09.800 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:11:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:11:09.800 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:11:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:11:09.800 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:11:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:11:09.800 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:11:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:11:09.800 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:11:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:11:09.800 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:11:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:11:09.800 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:11:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:11:09.800 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:11:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:11:09.800 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:11:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:11:09.800 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:11:09.800 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:11:09.800 139328 ERROR oslo_service.periodic_task >2018-06-24 20:11:39.671 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:11:39.671 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:11:39.678 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:11:39.679 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:11:39.714 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:11:39.714 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:11:39.714 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:11:39.714 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:11:39.714 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:11:39.714 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:11:39.714 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:11:39.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:11:39.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:11:39.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:11:49.717 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:11:49.751 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:11:49.751 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:11:49.751 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:11:49.751 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:11:49.751 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:11:49.751 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:11:49.751 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:11:49.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:11:49.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:11:49.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:12:09.754 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:12:09.788 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:12:09.788 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:12:09.788 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:12:09.788 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:12:09.788 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:12:09.788 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:12:09.788 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:12:09.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:12:09.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:12:09.790 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:12:09.790 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:12:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:12:09.790 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:12:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:12:09.790 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:12:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:12:09.790 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:12:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:12:09.790 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:12:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:12:09.790 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:12:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:12:09.790 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:12:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:12:09.790 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:12:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:12:09.790 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:12:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:12:09.790 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:12:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:12:09.790 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:12:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:12:09.790 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:12:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:12:09.790 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:12:09.790 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:12:09.790 139328 ERROR oslo_service.periodic_task >2018-06-24 20:12:39.672 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:12:39.673 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:12:39.679 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:12:39.680 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:12:39.715 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:12:39.715 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:12:39.715 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:12:39.715 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:12:39.715 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:12:39.715 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:12:39.715 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:12:39.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:12:39.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:12:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:12:49.718 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:12:49.750 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:12:49.750 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:12:49.750 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:12:49.750 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:12:49.750 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:12:49.750 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:12:49.750 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:12:49.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:12:49.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:12:49.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:13:09.754 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:13:09.785 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:13:09.785 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:13:09.785 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:13:09.785 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:13:09.785 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:13:09.785 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:13:09.785 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:13:09.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:13:09.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.107 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:13:09.788 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:13:09.788 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:13:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:13:09.788 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:13:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:13:09.788 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:13:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:13:09.788 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:13:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:13:09.788 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:13:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:13:09.788 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:13:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:13:09.788 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:13:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:13:09.788 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:13:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:13:09.788 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:13:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:13:09.788 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:13:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:13:09.788 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:13:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:13:09.788 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:13:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:13:09.788 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:13:09.788 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:13:09.788 139328 ERROR oslo_service.periodic_task >2018-06-24 20:13:39.669 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:13:39.670 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:13:39.678 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:13:39.679 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:13:39.721 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:13:39.721 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:13:39.721 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:13:39.721 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:13:39.721 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:13:39.721 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:13:39.721 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:13:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:13:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.044 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:13:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:13:49.725 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:13:49.767 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:13:49.767 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:13:49.767 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:13:49.767 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:13:49.767 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:13:49.767 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:13:49.767 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:13:49.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:13:49.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.09 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:13:49.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:14:09.771 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:14:09.804 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:14:09.804 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:14:09.804 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:14:09.804 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:14:09.804 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:14:09.804 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:14:09.804 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:14:09.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:14:09.807 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.127 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:14:09.807 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:14:09.807 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:14:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:14:09.807 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:14:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:14:09.807 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:14:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:14:09.807 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:14:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:14:09.807 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:14:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:14:09.807 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:14:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:14:09.807 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:14:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:14:09.807 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:14:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:14:09.807 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:14:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:14:09.807 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:14:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:14:09.807 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:14:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:14:09.807 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:14:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:14:09.807 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:14:09.807 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:14:09.807 139328 ERROR oslo_service.periodic_task >2018-06-24 20:14:39.668 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:14:39.670 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:14:39.676 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:14:39.677 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:14:39.711 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:14:39.711 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:14:39.711 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:14:39.711 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:14:39.711 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:14:39.711 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:14:39.711 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:14:39.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:14:39.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:14:39.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:14:49.714 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:14:49.747 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:14:49.747 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:14:49.747 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:14:49.747 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:14:49.747 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:14:49.747 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:14:49.747 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:14:49.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:14:49.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:14:49.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:15:09.750 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:15:09.783 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:15:09.783 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:15:09.783 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:15:09.783 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:15:09.783 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:15:09.783 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:15:09.783 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:15:09.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:15:09.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:15:09.786 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:15:09.786 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:15:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:15:09.786 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:15:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:15:09.786 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:15:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:15:09.786 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:15:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:15:09.786 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:15:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:15:09.786 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:15:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:15:09.786 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:15:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:15:09.786 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:15:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:15:09.786 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:15:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:15:09.786 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:15:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:15:09.786 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:15:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:15:09.786 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:15:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:15:09.786 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:15:09.786 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:15:09.786 139328 ERROR oslo_service.periodic_task >2018-06-24 20:15:39.667 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:15:39.668 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:15:39.675 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:15:39.676 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:15:39.710 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:15:39.710 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:15:39.710 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:15:39.710 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:15:39.710 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:15:39.710 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:15:39.710 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:15:39.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:15:39.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:15:39.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:15:49.713 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:15:49.748 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:15:49.748 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:15:49.748 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:15:49.748 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:15:49.748 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:15:49.748 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:15:49.748 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:15:49.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:15:49.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:15:49.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:16:09.751 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:16:09.783 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:16:09.783 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:16:09.783 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:16:09.783 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:16:09.783 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:16:09.783 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:16:09.783 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:16:09.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:16:09.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:16:09.786 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:16:09.786 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:16:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:16:09.786 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:16:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:16:09.786 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:16:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:16:09.786 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:16:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:16:09.786 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:16:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:16:09.786 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:16:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:16:09.786 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:16:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:16:09.786 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:16:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:16:09.786 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:16:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:16:09.786 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:16:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:16:09.786 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:16:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:16:09.786 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:16:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:16:09.786 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:16:09.786 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:16:09.786 139328 ERROR oslo_service.periodic_task >2018-06-24 20:16:39.667 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:16:39.668 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:16:39.676 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:16:39.677 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:16:39.710 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:16:39.710 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:16:39.710 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:16:39.710 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:16:39.710 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:16:39.710 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:16:39.710 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:16:39.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:16:39.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:16:39.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:16:49.713 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:16:49.747 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:16:49.747 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:16:49.747 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:16:49.747 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:16:49.747 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:16:49.747 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:16:49.747 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:16:49.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:16:49.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:16:49.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:17:09.750 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:17:09.782 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:17:09.782 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:17:09.782 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:17:09.782 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:17:09.782 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:17:09.782 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:17:09.782 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:17:09.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:17:09.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:17:09.785 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:17:09.785 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:17:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:17:09.785 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:17:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:17:09.785 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:17:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:17:09.785 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:17:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:17:09.785 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:17:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:17:09.785 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:17:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:17:09.785 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:17:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:17:09.785 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:17:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:17:09.785 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:17:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:17:09.785 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:17:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:17:09.785 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:17:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:17:09.785 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:17:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:17:09.785 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:17:09.785 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:17:09.785 139328 ERROR oslo_service.periodic_task >2018-06-24 20:17:39.666 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:17:39.668 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:17:39.674 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:17:39.675 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:17:39.710 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:17:39.710 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:17:39.710 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:17:39.710 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:17:39.710 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:17:39.710 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:17:39.710 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:17:39.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:17:39.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:17:39.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:17:49.713 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:17:49.746 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:17:49.746 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:17:49.746 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:17:49.746 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:17:49.746 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:17:49.746 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:17:49.746 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:17:49.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:17:49.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:17:49.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:18:09.749 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:18:09.784 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:18:09.784 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:18:09.784 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:18:09.784 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:18:09.784 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:18:09.784 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:18:09.784 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:18:09.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:18:09.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:18:09.787 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:18:09.787 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:18:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:18:09.787 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:18:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:18:09.787 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:18:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:18:09.787 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:18:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:18:09.787 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:18:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:18:09.787 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:18:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:18:09.787 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:18:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:18:09.787 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:18:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:18:09.787 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:18:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:18:09.787 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:18:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:18:09.787 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:18:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:18:09.787 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:18:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:18:09.787 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:18:09.787 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:18:09.787 139328 ERROR oslo_service.periodic_task >2018-06-24 20:18:39.668 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:18:39.669 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:18:39.680 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:18:39.681 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:18:39.716 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:18:39.716 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:18:39.716 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:18:39.716 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:18:39.716 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:18:39.716 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:18:39.716 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:18:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:18:39.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:18:39.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:18:49.720 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:18:49.754 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:18:49.754 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:18:49.754 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:18:49.754 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:18:49.754 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:18:49.754 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:18:49.754 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:18:49.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:18:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:18:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:19:09.758 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:19:09.798 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:19:09.798 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:19:09.798 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:19:09.798 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:19:09.798 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:19:09.798 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:19:09.798 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:19:09.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:19:09.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.119 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:19:09.800 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:19:09.800 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:19:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:19:09.800 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:19:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:19:09.800 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:19:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:19:09.800 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:19:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:19:09.800 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:19:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:19:09.800 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:19:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:19:09.800 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:19:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:19:09.800 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:19:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:19:09.800 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:19:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:19:09.800 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:19:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:19:09.800 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:19:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:19:09.800 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:19:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:19:09.800 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:19:09.800 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:19:09.800 139328 ERROR oslo_service.periodic_task >2018-06-24 20:19:39.672 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:19:39.673 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:19:39.681 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:19:39.681 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:19:39.714 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:19:39.714 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:19:39.714 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:19:39.714 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:19:39.714 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:19:39.714 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:19:39.714 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:19:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:19:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:19:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:19:49.719 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:19:49.755 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:19:49.755 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:19:49.755 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:19:49.755 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:19:49.755 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:19:49.755 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:19:49.755 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:19:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:19:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:19:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:20:09.758 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:20:09.792 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:20:09.792 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:20:09.792 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:20:09.792 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:20:09.792 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:20:09.792 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:20:09.792 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:20:09.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:20:09.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:20:09.795 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:20:09.795 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:20:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:20:09.795 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:20:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:20:09.795 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:20:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:20:09.795 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:20:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:20:09.795 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:20:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:20:09.795 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:20:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:20:09.795 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:20:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:20:09.795 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:20:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:20:09.795 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:20:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:20:09.795 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:20:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:20:09.795 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:20:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:20:09.795 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:20:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:20:09.795 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:20:09.795 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:20:09.795 139328 ERROR oslo_service.periodic_task >2018-06-24 20:20:39.676 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:20:39.677 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:20:39.683 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:20:39.684 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:20:39.717 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:20:39.717 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:20:39.717 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:20:39.717 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:20:39.717 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:20:39.717 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:20:39.717 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:20:39.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:20:39.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:20:39.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:20:49.720 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:20:49.754 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:20:49.754 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:20:49.754 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:20:49.754 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:20:49.754 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:20:49.754 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:20:49.754 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:20:49.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:20:49.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:20:49.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:21:09.757 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:21:09.790 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:21:09.790 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:21:09.790 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:21:09.790 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:21:09.790 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:21:09.790 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:21:09.790 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:21:09.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:21:09.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:21:09.793 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:21:09.793 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:21:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:21:09.793 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:21:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:21:09.793 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:21:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:21:09.793 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:21:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:21:09.793 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:21:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:21:09.793 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:21:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:21:09.793 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:21:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:21:09.793 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:21:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:21:09.793 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:21:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:21:09.793 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:21:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:21:09.793 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:21:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:21:09.793 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:21:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:21:09.793 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:21:09.793 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:21:09.793 139328 ERROR oslo_service.periodic_task >2018-06-24 20:21:39.674 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:21:39.676 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:21:39.684 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:21:39.684 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:21:39.724 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:21:39.724 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:21:39.724 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:21:39.724 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:21:39.724 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:21:39.724 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:21:39.724 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:21:39.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:21:39.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.042 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:21:39.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:21:49.728 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:21:49.763 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:21:49.763 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:21:49.763 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:21:49.763 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:21:49.763 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:21:49.763 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:21:49.763 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:21:49.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:21:49.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.08 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:21:49.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:22:09.766 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:22:09.800 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:22:09.800 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:22:09.800 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:22:09.800 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:22:09.800 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:22:09.800 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:22:09.800 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:22:09.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:22:09.803 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.118 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:22:09.803 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:22:09.803 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:22:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:22:09.803 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:22:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:22:09.803 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:22:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:22:09.803 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:22:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:22:09.803 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:22:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:22:09.803 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:22:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:22:09.803 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:22:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:22:09.803 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:22:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:22:09.803 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:22:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:22:09.803 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:22:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:22:09.803 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:22:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:22:09.803 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:22:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:22:09.803 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:22:09.803 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:22:09.803 139328 ERROR oslo_service.periodic_task >2018-06-24 20:22:39.674 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:22:39.675 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:22:39.682 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:22:39.682 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:22:39.719 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:22:39.719 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:22:39.719 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:22:39.719 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:22:39.719 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:22:39.719 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:22:39.719 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:22:39.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:22:39.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:22:39.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:22:49.722 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:22:49.755 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:22:49.755 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:22:49.755 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:22:49.755 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:22:49.755 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:22:49.755 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:22:49.755 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:22:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:22:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:22:49.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:23:09.758 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:23:09.792 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:23:09.792 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:23:09.792 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:23:09.792 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:23:09.792 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:23:09.792 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:23:09.792 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:23:09.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:23:09.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:23:09.794 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:23:09.794 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:23:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:23:09.794 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:23:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:23:09.794 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:23:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:23:09.794 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:23:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:23:09.794 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:23:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:23:09.794 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:23:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:23:09.794 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:23:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:23:09.794 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:23:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:23:09.794 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:23:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:23:09.794 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:23:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:23:09.794 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:23:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:23:09.794 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:23:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:23:09.794 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:23:09.794 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:23:09.794 139328 ERROR oslo_service.periodic_task >2018-06-24 20:23:39.676 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:23:39.677 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:23:39.685 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:23:39.685 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:23:39.721 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:23:39.721 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:23:39.721 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:23:39.721 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:23:39.721 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:23:39.721 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:23:39.721 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:23:39.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:23:39.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:23:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:23:49.724 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:23:49.759 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:23:49.759 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:23:49.759 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:23:49.759 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:23:49.759 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:23:49.759 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:23:49.759 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:23:49.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:23:49.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:23:49.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:24:09.762 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:24:09.794 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:24:09.794 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:24:09.794 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:24:09.794 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:24:09.794 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:24:09.794 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:24:09.794 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:24:09.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:24:09.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:24:09.796 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:24:09.796 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:24:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:24:09.796 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:24:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:24:09.796 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:24:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:24:09.796 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:24:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:24:09.796 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:24:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:24:09.796 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:24:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:24:09.796 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:24:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:24:09.796 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:24:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:24:09.796 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:24:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:24:09.796 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:24:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:24:09.796 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:24:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:24:09.796 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:24:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:24:09.796 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:24:09.796 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:24:09.796 139328 ERROR oslo_service.periodic_task >2018-06-24 20:24:39.678 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:24:39.679 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:24:39.686 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:24:39.686 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:24:39.719 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:24:39.719 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:24:39.719 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:24:39.719 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:24:39.719 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:24:39.719 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:24:39.719 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:24:39.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:24:39.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:24:39.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:24:49.722 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:24:49.763 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:24:49.763 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:24:49.763 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:24:49.763 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:24:49.763 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:24:49.763 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:24:49.763 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:24:49.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:24:49.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:24:49.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:25:09.767 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:25:09.800 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:25:09.800 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:25:09.800 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:25:09.800 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:25:09.800 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:25:09.800 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:25:09.800 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:25:09.803 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:25:09.803 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:25:09.803 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:25:09.803 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:25:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:25:09.803 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:25:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:25:09.803 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:25:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:25:09.803 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:25:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:25:09.803 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:25:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:25:09.803 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:25:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:25:09.803 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:25:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:25:09.803 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:25:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:25:09.803 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:25:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:25:09.803 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:25:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:25:09.803 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:25:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:25:09.803 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:25:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:25:09.803 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:25:09.803 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:25:09.803 139328 ERROR oslo_service.periodic_task >2018-06-24 20:25:39.675 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:25:39.676 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:25:39.683 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:25:39.684 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:25:39.717 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:25:39.717 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:25:39.717 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:25:39.717 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:25:39.717 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:25:39.717 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:25:39.717 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:25:39.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:25:39.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:25:39.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:25:49.721 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:25:49.755 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:25:49.755 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:25:49.755 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:25:49.755 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:25:49.755 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:25:49.755 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:25:49.755 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:25:49.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:25:49.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:25:49.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:26:09.759 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:26:09.793 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:26:09.793 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:26:09.793 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:26:09.793 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:26:09.793 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:26:09.793 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:26:09.793 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:26:09.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:26:09.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:26:09.796 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:26:09.796 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:26:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:26:09.796 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:26:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:26:09.796 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:26:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:26:09.796 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:26:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:26:09.796 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:26:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:26:09.796 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:26:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:26:09.796 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:26:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:26:09.796 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:26:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:26:09.796 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:26:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:26:09.796 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:26:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:26:09.796 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:26:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:26:09.796 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:26:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:26:09.796 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:26:09.796 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:26:09.796 139328 ERROR oslo_service.periodic_task >2018-06-24 20:26:39.677 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:26:39.678 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:26:39.687 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:26:39.688 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:26:39.724 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:26:39.724 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:26:39.724 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:26:39.724 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:26:39.724 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:26:39.724 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:26:39.724 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:26:39.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:26:39.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:26:39.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:26:49.727 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:26:49.760 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:26:49.760 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:26:49.760 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:26:49.760 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:26:49.760 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:26:49.760 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:26:49.760 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:26:49.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:26:49.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:26:49.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:27:09.764 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:27:09.796 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:27:09.796 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:27:09.796 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:27:09.796 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:27:09.796 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:27:09.796 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:27:09.796 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:27:09.798 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:27:09.799 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:27:09.799 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:27:09.799 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:27:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:27:09.799 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:27:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:27:09.799 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:27:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:27:09.799 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:27:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:27:09.799 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:27:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:27:09.799 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:27:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:27:09.799 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:27:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:27:09.799 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:27:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:27:09.799 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:27:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:27:09.799 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:27:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:27:09.799 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:27:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:27:09.799 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:27:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:27:09.799 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:27:09.799 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:27:09.799 139328 ERROR oslo_service.periodic_task >2018-06-24 20:27:39.681 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:27:39.682 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:27:39.690 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:27:39.691 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:27:39.727 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:27:39.727 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:27:39.727 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:27:39.727 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:27:39.727 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:27:39.727 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:27:39.727 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:27:39.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:27:39.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:27:39.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:27:49.730 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:27:49.771 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:27:49.771 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:27:49.771 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:27:49.771 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:27:49.771 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:27:49.771 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:27:49.771 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:27:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:27:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.083 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:27:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:28:09.775 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:28:09.810 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:28:09.810 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:28:09.810 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:28:09.810 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:28:09.810 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:28:09.810 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:28:09.810 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:28:09.812 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:28:09.813 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.122 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:28:09.813 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:28:09.813 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:28:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:28:09.813 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:28:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:28:09.813 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:28:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:28:09.813 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:28:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:28:09.813 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:28:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:28:09.813 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:28:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:28:09.813 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:28:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:28:09.813 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:28:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:28:09.813 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:28:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:28:09.813 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:28:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:28:09.813 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:28:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:28:09.813 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:28:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:28:09.813 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:28:09.813 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:28:09.813 139328 ERROR oslo_service.periodic_task >2018-06-24 20:28:39.685 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:28:39.686 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:28:39.695 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:28:39.695 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:28:39.730 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:28:39.730 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:28:39.730 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:28:39.730 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:28:39.730 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:28:39.730 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:28:39.730 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:28:39.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:28:39.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:28:39.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:28:49.734 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:28:49.766 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:28:49.766 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:28:49.766 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:28:49.766 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:28:49.766 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:28:49.766 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:28:49.766 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:28:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:28:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:28:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:29:09.770 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:29:09.803 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:29:09.803 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:29:09.803 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:29:09.803 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:29:09.803 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:29:09.803 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:29:09.803 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:29:09.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:29:09.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:29:09.806 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:29:09.806 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:29:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:29:09.806 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:29:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:29:09.806 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:29:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:29:09.806 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:29:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:29:09.806 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:29:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:29:09.806 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:29:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:29:09.806 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:29:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:29:09.806 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:29:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:29:09.806 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:29:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:29:09.806 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:29:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:29:09.806 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:29:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:29:09.806 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:29:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:29:09.806 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:29:09.806 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:29:09.806 139328 ERROR oslo_service.periodic_task >2018-06-24 20:29:39.688 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:29:39.688 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:29:39.695 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:29:39.696 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:29:39.727 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:29:39.727 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:29:39.727 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:29:39.727 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:29:39.727 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:29:39.727 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:29:39.727 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:29:39.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:29:39.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:29:39.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:29:49.730 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:29:49.768 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:29:49.768 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:29:49.768 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:29:49.768 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:29:49.768 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:29:49.768 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:29:49.768 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:29:49.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:29:49.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:29:49.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:30:09.771 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:30:09.806 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:30:09.806 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:30:09.806 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:30:09.806 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:30:09.806 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:30:09.806 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:30:09.806 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:30:09.809 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:30:09.809 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:30:09.810 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:30:09.810 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:30:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:30:09.810 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:30:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:30:09.810 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:30:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:30:09.810 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:30:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:30:09.810 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:30:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:30:09.810 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:30:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:30:09.810 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:30:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:30:09.810 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:30:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:30:09.810 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:30:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:30:09.810 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:30:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:30:09.810 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:30:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:30:09.810 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:30:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:30:09.810 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:30:09.810 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:30:09.810 139328 ERROR oslo_service.periodic_task >2018-06-24 20:30:39.691 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:30:39.693 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:30:39.700 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:30:39.701 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:30:39.734 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:30:39.734 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:30:39.734 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:30:39.734 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:30:39.734 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:30:39.734 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:30:39.734 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:30:39.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:30:39.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:30:39.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:30:49.738 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:30:49.782 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:30:49.782 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:30:49.782 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:30:49.782 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:30:49.782 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:30:49.782 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:30:49.782 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:30:49.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:30:49.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.084 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:30:49.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:31:09.786 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:31:09.819 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:31:09.819 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:31:09.819 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:31:09.819 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:31:09.819 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:31:09.819 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:31:09.819 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:31:09.821 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:31:09.821 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.121 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:31:09.822 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:31:09.822 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:31:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:31:09.822 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:31:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:31:09.822 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:31:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:31:09.822 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:31:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:31:09.822 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:31:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:31:09.822 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:31:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:31:09.822 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:31:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:31:09.822 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:31:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:31:09.822 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:31:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:31:09.822 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:31:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:31:09.822 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:31:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:31:09.822 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:31:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:31:09.822 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:31:09.822 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:31:09.822 139328 ERROR oslo_service.periodic_task >2018-06-24 20:31:39.693 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:31:39.694 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:32:39.692 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:32:39.693 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:32:39.700 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:32:39.701 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:32:39.736 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:32:39.736 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:32:39.736 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:32:39.736 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:32:39.736 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:32:39.736 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:32:39.736 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:32:39.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:32:39.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:32:39.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:32:49.740 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:32:49.774 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:32:49.774 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:32:49.774 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:32:49.774 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:32:49.774 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:32:49.774 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:32:49.774 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:32:49.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:32:49.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:32:49.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:33:09.777 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:33:09.813 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:33:09.813 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:33:09.813 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:33:09.813 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:33:09.813 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:33:09.813 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:33:09.813 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:33:09.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:33:09.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:33:09.816 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:33:09.816 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:33:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:33:09.816 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:33:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:33:09.816 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:33:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:33:09.816 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:33:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:33:09.816 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:33:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:33:09.816 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:33:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:33:09.816 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:33:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:33:09.816 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:33:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:33:09.816 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:33:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:33:09.816 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:33:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:33:09.816 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:33:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:33:09.816 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:33:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:33:09.816 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:33:09.816 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:33:09.816 139328 ERROR oslo_service.periodic_task >2018-06-24 20:33:39.687 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:33:39.689 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:33:39.705 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:33:39.706 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:33:39.742 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:33:39.742 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:33:39.742 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:33:39.742 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:33:39.742 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:33:39.742 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:33:39.742 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:33:39.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:33:39.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:33:39.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:33:49.745 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:33:49.777 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:33:49.777 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:33:49.777 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:33:49.777 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:33:49.777 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:33:49.777 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:33:49.777 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:33:49.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:33:49.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:33:49.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:34:09.780 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:34:09.813 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:34:09.813 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:34:09.813 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:34:09.813 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:34:09.813 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:34:09.813 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:34:09.813 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:34:09.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:34:09.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:34:09.816 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:34:09.816 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:34:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:34:09.816 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:34:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:34:09.816 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:34:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:34:09.816 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:34:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:34:09.816 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:34:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:34:09.816 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:34:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:34:09.816 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:34:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:34:09.816 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:34:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:34:09.816 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:34:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:34:09.816 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:34:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:34:09.816 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:34:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:34:09.816 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:34:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:34:09.816 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:34:09.816 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:34:09.816 139328 ERROR oslo_service.periodic_task >2018-06-24 20:34:39.687 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:34:39.688 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:34:39.697 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:34:39.698 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:34:39.735 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:34:39.735 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:34:39.735 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:34:39.735 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:34:39.735 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:34:39.735 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:34:39.735 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:34:39.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:34:39.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:34:39.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:34:49.739 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:34:49.773 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:34:49.773 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:34:49.773 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:34:49.773 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:34:49.773 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:34:49.773 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:34:49.773 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:34:49.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:34:49.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:34:49.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:35:09.776 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:35:09.810 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:35:09.810 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:35:09.810 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:35:09.810 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:35:09.810 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:35:09.810 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:35:09.810 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:35:09.812 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:35:09.812 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:35:09.812 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:35:09.812 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:35:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:35:09.812 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:35:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:35:09.812 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:35:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:35:09.812 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:35:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:35:09.812 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:35:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:35:09.812 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:35:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:35:09.812 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:35:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:35:09.812 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:35:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:35:09.812 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:35:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:35:09.812 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:35:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:35:09.812 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:35:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:35:09.812 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:35:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:35:09.812 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:35:09.812 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:35:09.812 139328 ERROR oslo_service.periodic_task >2018-06-24 20:35:39.684 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:35:39.685 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:35:39.694 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:35:39.695 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:35:39.729 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:35:39.729 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:35:39.729 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:35:39.729 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:35:39.729 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:35:39.729 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:35:39.729 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:35:39.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:35:39.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:35:39.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:35:49.732 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:35:49.769 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:35:49.769 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:35:49.769 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:35:49.769 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:35:49.769 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:35:49.769 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:35:49.769 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:35:49.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:35:49.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:35:49.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:36:09.772 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:36:09.805 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:36:09.805 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:36:09.805 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:36:09.805 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:36:09.805 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:36:09.805 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:36:09.805 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:36:09.807 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:36:09.807 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:36:09.808 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:36:09.808 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:36:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:36:09.808 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:36:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:36:09.808 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:36:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:36:09.808 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:36:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:36:09.808 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:36:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:36:09.808 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:36:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:36:09.808 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:36:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:36:09.808 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:36:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:36:09.808 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:36:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:36:09.808 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:36:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:36:09.808 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:36:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:36:09.808 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:36:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:36:09.808 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:36:09.808 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:36:09.808 139328 ERROR oslo_service.periodic_task >2018-06-24 20:36:39.689 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:36:39.690 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:36:39.697 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:36:39.697 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:36:39.730 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:36:39.730 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:36:39.730 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:36:39.730 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:36:39.730 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:36:39.730 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:36:39.730 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:36:39.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:36:39.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:36:39.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:36:49.733 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:36:49.770 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:36:49.770 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:36:49.770 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:36:49.770 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:36:49.770 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:36:49.770 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:36:49.770 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:36:49.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:36:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:36:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:37:09.774 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:37:09.808 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:37:09.808 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:37:09.808 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:37:09.808 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:37:09.808 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:37:09.808 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:37:09.808 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:37:09.811 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:37:09.811 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:37:09.811 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:37:09.811 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:37:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:37:09.811 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:37:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:37:09.811 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:37:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:37:09.811 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:37:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:37:09.811 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:37:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:37:09.811 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:37:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:37:09.811 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:37:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:37:09.811 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:37:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:37:09.811 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:37:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:37:09.811 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:37:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:37:09.811 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:37:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:37:09.811 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:37:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:37:09.811 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:37:09.811 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:37:09.811 139328 ERROR oslo_service.periodic_task >2018-06-24 20:37:39.693 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:37:39.694 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:37:39.701 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:37:39.702 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:37:39.740 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:37:39.740 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:37:39.740 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:37:39.740 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:37:39.740 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:37:39.740 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:37:39.740 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:37:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:37:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.041 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:37:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:37:49.744 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:37:49.777 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:37:49.777 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:37:49.777 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:37:49.777 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:37:49.777 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:37:49.777 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:37:49.777 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:37:49.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:37:49.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:37:49.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:38:09.780 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:38:09.813 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:38:09.813 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:38:09.813 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:38:09.813 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:38:09.813 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:38:09.813 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:38:09.813 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:38:09.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:38:09.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:38:09.816 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:38:09.816 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:38:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:38:09.816 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:38:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:38:09.816 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:38:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:38:09.816 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:38:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:38:09.816 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:38:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:38:09.816 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:38:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:38:09.816 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:38:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:38:09.816 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:38:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:38:09.816 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:38:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:38:09.816 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:38:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:38:09.816 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:38:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:38:09.816 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:38:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:38:09.816 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:38:09.816 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:38:09.816 139328 ERROR oslo_service.periodic_task >2018-06-24 20:38:39.697 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:38:39.698 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:38:39.711 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:38:39.711 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:38:39.743 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:38:39.743 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:38:39.743 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:38:39.743 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:38:39.743 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:38:39.743 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:38:39.743 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:38:39.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:38:39.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.033 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:38:39.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:38:49.746 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:38:49.780 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:38:49.780 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:38:49.780 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:38:49.780 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:38:49.780 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:38:49.780 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:38:49.780 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:38:49.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:38:49.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:38:49.783 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:39:09.783 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:39:09.816 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:39:09.816 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:39:09.816 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:39:09.816 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:39:09.816 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:39:09.816 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:39:09.816 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:39:09.818 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:39:09.819 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.107 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:39:09.819 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:39:09.819 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:39:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:39:09.819 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:39:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:39:09.819 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:39:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:39:09.819 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:39:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:39:09.819 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:39:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:39:09.819 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:39:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:39:09.819 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:39:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:39:09.819 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:39:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:39:09.819 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:39:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:39:09.819 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:39:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:39:09.819 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:39:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:39:09.819 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:39:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:39:09.819 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:39:09.819 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:39:09.819 139328 ERROR oslo_service.periodic_task >2018-06-24 20:39:39.700 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:39:39.702 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:39:39.708 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:39:39.709 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:39:39.743 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:39:39.743 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:39:39.743 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:39:39.743 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:39:39.743 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:39:39.743 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:39:39.743 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:39:39.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:39:39.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:39:39.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:39:49.747 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:39:49.780 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:39:49.780 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:39:49.780 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:39:49.780 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:39:49.780 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:39:49.780 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:39:49.780 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:39:49.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:39:49.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:39:49.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:40:09.783 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:40:09.817 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:40:09.817 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:40:09.817 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:40:09.817 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:40:09.817 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:40:09.817 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:40:09.817 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:40:09.819 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:40:09.819 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:40:09.820 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:40:09.820 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:40:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:40:09.820 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:40:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:40:09.820 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:40:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:40:09.820 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:40:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:40:09.820 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:40:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:40:09.820 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:40:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:40:09.820 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:40:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:40:09.820 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:40:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:40:09.820 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:40:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:40:09.820 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:40:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:40:09.820 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:40:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:40:09.820 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:40:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:40:09.820 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:40:09.820 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:40:09.820 139328 ERROR oslo_service.periodic_task >2018-06-24 20:40:39.701 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:40:39.702 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:40:39.710 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:40:39.710 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:40:39.743 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:40:39.743 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:40:39.743 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:40:39.743 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:40:39.743 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:40:39.743 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:40:39.743 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:40:39.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:40:39.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:40:39.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:40:49.747 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:40:49.780 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:40:49.780 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:40:49.780 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:40:49.780 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:40:49.780 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:40:49.780 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:40:49.780 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:40:49.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:40:49.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:40:49.783 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:41:09.783 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:41:09.816 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:41:09.816 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:41:09.816 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:41:09.816 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:41:09.816 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:41:09.816 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:41:09.816 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:41:09.819 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:41:09.819 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:41:09.819 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:41:09.819 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:41:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:41:09.819 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:41:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:41:09.819 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:41:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:41:09.819 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:41:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:41:09.819 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:41:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:41:09.819 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:41:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:41:09.819 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:41:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:41:09.819 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:41:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:41:09.819 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:41:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:41:09.819 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:41:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:41:09.819 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:41:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:41:09.819 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:41:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:41:09.819 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:41:09.819 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:41:09.819 139328 ERROR oslo_service.periodic_task >2018-06-24 20:41:39.701 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:41:39.702 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:41:39.716 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:41:39.716 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:41:39.750 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:41:39.750 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:41:39.750 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:41:39.750 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:41:39.750 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:41:39.750 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:41:39.750 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:41:39.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:41:39.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:41:39.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:41:49.753 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:41:49.788 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:41:49.788 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:41:49.788 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:41:49.788 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:41:49.788 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:41:49.788 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:41:49.788 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:41:49.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:41:49.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:41:49.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:42:09.792 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:42:09.826 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:42:09.826 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:42:09.826 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:42:09.826 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:42:09.826 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:42:09.826 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:42:09.826 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:42:09.828 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:42:09.828 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:42:09.828 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:42:09.828 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:42:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:42:09.828 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:42:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:42:09.828 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:42:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:42:09.828 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:42:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:42:09.828 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:42:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:42:09.828 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:42:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:42:09.828 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:42:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:42:09.828 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:42:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:42:09.828 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:42:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:42:09.828 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:42:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:42:09.828 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:42:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:42:09.828 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:42:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:42:09.828 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:42:09.828 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:42:09.828 139328 ERROR oslo_service.periodic_task >2018-06-24 20:42:39.700 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:42:39.701 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:42:39.708 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:42:39.709 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:42:39.747 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:42:39.747 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:42:39.747 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:42:39.747 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:42:39.747 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:42:39.747 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:42:39.747 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:42:39.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:42:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.041 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:42:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:42:49.751 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:42:49.796 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:42:49.796 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:42:49.796 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:42:49.796 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:42:49.796 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:42:49.796 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:42:49.796 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:42:49.799 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:42:49.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.091 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:42:49.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:43:09.801 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:43:09.837 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:43:09.837 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:43:09.837 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:43:09.837 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:43:09.837 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:43:09.837 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:43:09.837 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:43:09.839 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:43:09.839 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.131 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:43:09.840 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:43:09.840 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:43:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:43:09.840 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:43:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:43:09.840 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:43:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:43:09.840 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:43:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:43:09.840 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:43:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:43:09.840 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:43:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:43:09.840 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:43:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:43:09.840 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:43:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:43:09.840 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:43:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:43:09.840 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:43:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:43:09.840 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:43:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:43:09.840 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:43:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:43:09.840 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:43:09.840 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:43:09.840 139328 ERROR oslo_service.periodic_task >2018-06-24 20:43:39.701 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:43:39.702 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:43:39.713 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:43:39.714 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:43:39.747 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:43:39.747 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:43:39.747 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:43:39.747 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:43:39.747 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:43:39.747 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:43:39.747 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:43:39.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:43:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:43:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:43:49.751 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:43:49.788 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:43:49.788 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:43:49.788 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:43:49.788 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:43:49.788 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:43:49.788 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:43:49.788 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:43:49.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:43:49.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:43:49.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:44:09.791 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:44:09.824 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:44:09.824 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:44:09.824 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:44:09.824 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:44:09.824 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:44:09.824 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:44:09.824 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:44:09.826 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:44:09.827 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:44:09.827 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:44:09.827 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:44:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:44:09.827 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:44:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:44:09.827 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:44:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:44:09.827 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:44:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:44:09.827 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:44:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:44:09.827 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:44:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:44:09.827 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:44:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:44:09.827 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:44:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:44:09.827 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:44:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:44:09.827 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:44:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:44:09.827 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:44:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:44:09.827 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:44:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:44:09.827 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:44:09.827 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:44:09.827 139328 ERROR oslo_service.periodic_task >2018-06-24 20:44:39.698 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:44:39.699 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:44:39.706 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:44:39.707 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:44:39.740 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:44:39.740 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:44:39.740 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:44:39.740 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:44:39.740 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:44:39.740 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:44:39.740 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:44:39.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:44:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:44:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:44:49.744 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:44:49.779 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:44:49.779 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:44:49.779 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:44:49.779 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:44:49.779 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:44:49.779 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:44:49.779 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:44:49.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:44:49.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:44:49.783 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:45:09.783 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:45:09.820 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:45:09.820 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:45:09.820 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:45:09.820 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:45:09.820 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:45:09.820 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:45:09.820 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:45:09.822 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:45:09.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:45:09.823 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:45:09.823 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:45:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:45:09.823 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:45:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:45:09.823 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:45:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:45:09.823 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:45:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:45:09.823 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:45:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:45:09.823 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:45:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:45:09.823 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:45:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:45:09.823 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:45:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:45:09.823 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:45:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:45:09.823 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:45:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:45:09.823 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:45:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:45:09.823 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:45:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:45:09.823 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:45:09.823 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:45:09.823 139328 ERROR oslo_service.periodic_task >2018-06-24 20:45:39.695 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:45:39.697 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:45:39.703 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:45:39.703 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:45:39.736 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:45:39.736 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:45:39.736 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:45:39.736 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:45:39.736 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:45:39.736 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:45:39.736 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:45:39.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:45:39.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:45:39.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:45:49.739 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:45:49.772 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:45:49.772 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:45:49.772 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:45:49.772 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:45:49.772 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:45:49.772 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:45:49.772 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:45:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:45:49.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:45:49.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:46:09.775 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:46:09.809 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:46:09.809 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:46:09.809 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:46:09.809 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:46:09.809 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:46:09.809 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:46:09.809 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:46:09.812 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:46:09.812 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:46:09.812 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:46:09.812 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:46:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:46:09.812 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:46:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:46:09.812 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:46:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:46:09.812 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:46:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:46:09.812 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:46:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:46:09.812 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:46:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:46:09.812 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:46:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:46:09.812 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:46:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:46:09.812 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:46:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:46:09.812 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:46:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:46:09.812 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:46:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:46:09.812 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:46:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:46:09.812 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:46:09.812 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:46:09.812 139328 ERROR oslo_service.periodic_task >2018-06-24 20:46:39.694 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:46:39.695 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:46:39.702 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:46:39.702 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:46:39.737 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:46:39.737 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:46:39.737 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:46:39.737 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:46:39.737 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:46:39.737 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:46:39.737 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:46:39.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:46:39.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:46:39.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:46:49.741 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:46:49.773 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:46:49.773 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:46:49.773 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:46:49.773 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:46:49.773 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:46:49.773 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:46:49.773 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:46:49.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:46:49.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:46:49.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:47:09.776 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:47:09.810 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:47:09.810 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:47:09.810 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:47:09.810 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:47:09.810 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:47:09.810 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:47:09.810 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:47:09.812 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:47:09.813 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:47:09.813 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:47:09.813 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:47:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:47:09.813 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:47:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:47:09.813 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:47:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:47:09.813 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:47:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:47:09.813 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:47:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:47:09.813 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:47:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:47:09.813 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:47:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:47:09.813 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:47:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:47:09.813 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:47:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:47:09.813 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:47:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:47:09.813 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:47:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:47:09.813 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:47:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:47:09.813 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:47:09.813 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:47:09.813 139328 ERROR oslo_service.periodic_task >2018-06-24 20:47:39.694 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:47:39.695 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:47:39.701 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:47:39.702 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:47:39.735 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:47:39.735 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:47:39.735 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:47:39.735 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:47:39.735 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:47:39.735 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:47:39.735 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:47:39.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:47:39.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:47:39.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:47:49.739 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:47:49.773 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:47:49.773 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:47:49.773 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:47:49.773 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:47:49.773 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:47:49.773 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:47:49.773 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:47:49.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:47:49.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:47:49.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:48:09.776 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:48:09.809 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:48:09.809 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:48:09.809 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:48:09.809 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:48:09.809 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:48:09.809 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:48:09.809 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:48:09.811 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:48:09.811 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:48:09.812 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:48:09.812 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:48:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:48:09.812 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:48:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:48:09.812 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:48:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:48:09.812 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:48:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:48:09.812 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:48:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:48:09.812 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:48:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:48:09.812 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:48:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:48:09.812 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:48:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:48:09.812 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:48:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:48:09.812 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:48:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:48:09.812 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:48:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:48:09.812 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:48:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:48:09.812 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:48:09.812 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:48:09.812 139328 ERROR oslo_service.periodic_task >2018-06-24 20:48:39.693 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:48:39.694 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:48:39.700 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:48:39.701 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:48:39.734 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:48:39.734 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:48:39.734 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:48:39.734 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:48:39.734 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:48:39.734 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:48:39.734 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:48:39.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:48:39.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:48:39.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:48:49.738 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:48:49.770 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:48:49.770 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:48:49.770 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:48:49.770 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:48:49.770 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:48:49.770 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:48:49.770 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:48:49.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:48:49.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:48:49.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:49:09.773 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:49:09.812 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:49:09.812 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:49:09.812 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:49:09.812 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:49:09.812 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:49:09.812 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:49:09.812 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:49:09.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:49:09.818 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:49:09.818 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:49:09.818 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:49:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:49:09.818 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:49:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:49:09.818 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:49:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:49:09.818 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:49:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:49:09.818 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:49:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:49:09.818 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:49:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:49:09.818 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:49:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:49:09.818 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:49:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:49:09.818 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:49:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:49:09.818 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:49:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:49:09.818 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:49:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:49:09.818 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:49:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:49:09.818 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:49:09.818 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:49:09.818 139328 ERROR oslo_service.periodic_task >2018-06-24 20:49:39.690 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:49:39.690 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:49:39.697 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:49:39.698 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:49:39.734 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:49:39.734 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:49:39.734 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:49:39.734 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:49:39.734 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:49:39.734 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:49:39.734 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:49:39.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:49:39.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:49:39.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:49:49.737 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:49:49.771 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:49:49.771 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:49:49.771 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:49:49.771 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:49:49.771 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:49:49.771 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:49:49.771 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:49:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:49:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:49:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:50:09.775 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:50:09.809 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:50:09.809 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:50:09.809 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:50:09.809 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:50:09.809 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:50:09.809 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:50:09.809 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:50:09.811 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:50:09.811 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:50:09.812 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:50:09.812 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:50:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:50:09.812 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:50:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:50:09.812 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:50:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:50:09.812 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:50:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:50:09.812 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:50:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:50:09.812 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:50:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:50:09.812 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:50:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:50:09.812 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:50:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:50:09.812 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:50:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:50:09.812 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:50:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:50:09.812 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:50:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:50:09.812 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:50:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:50:09.812 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:50:09.812 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:50:09.812 139328 ERROR oslo_service.periodic_task >2018-06-24 20:50:39.693 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:50:39.694 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:50:39.700 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:50:39.701 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:50:39.733 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:50:39.733 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:50:39.733 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:50:39.733 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:50:39.733 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:50:39.733 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:50:39.733 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:50:39.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:50:39.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:50:39.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:50:49.736 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:50:49.770 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:50:49.770 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:50:49.770 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:50:49.770 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:50:49.770 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:50:49.770 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:50:49.770 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:50:49.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:50:49.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:50:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:51:09.773 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:51:09.805 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:51:09.805 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:51:09.805 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:51:09.805 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:51:09.805 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:51:09.805 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:51:09.805 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:51:09.807 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:51:09.808 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.107 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:51:09.808 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:51:09.808 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:51:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:51:09.808 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:51:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:51:09.808 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:51:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:51:09.808 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:51:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:51:09.808 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:51:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:51:09.808 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:51:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:51:09.808 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:51:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:51:09.808 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:51:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:51:09.808 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:51:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:51:09.808 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:51:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:51:09.808 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:51:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:51:09.808 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:51:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:51:09.808 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:51:09.808 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:51:09.808 139328 ERROR oslo_service.periodic_task >2018-06-24 20:51:39.689 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:51:39.690 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:51:39.697 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:51:39.697 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:51:39.732 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:51:39.732 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:51:39.732 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:51:39.732 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:51:39.732 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:51:39.732 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:51:39.732 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:51:39.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:51:39.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:51:39.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:51:49.735 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:51:49.770 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:51:49.770 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:51:49.770 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:51:49.770 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:51:49.770 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:51:49.770 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:51:49.770 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:51:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:51:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:51:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:52:09.775 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:52:09.808 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:52:09.808 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:52:09.808 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:52:09.808 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:52:09.808 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:52:09.808 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:52:09.808 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:52:09.810 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:52:09.810 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:52:09.811 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:52:09.811 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:52:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:52:09.811 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:52:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:52:09.811 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:52:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:52:09.811 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:52:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:52:09.811 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:52:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:52:09.811 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:52:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:52:09.811 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:52:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:52:09.811 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:52:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:52:09.811 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:52:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:52:09.811 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:52:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:52:09.811 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:52:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:52:09.811 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:52:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:52:09.811 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:52:09.811 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:52:09.811 139328 ERROR oslo_service.periodic_task >2018-06-24 20:52:39.692 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:52:39.694 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:52:39.700 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:52:39.701 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:52:39.736 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:52:39.736 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:52:39.736 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:52:39.736 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:52:39.736 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:52:39.736 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:52:39.736 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:52:39.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:52:39.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:52:39.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:52:49.739 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:52:49.772 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:52:49.772 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:52:49.772 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:52:49.772 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:52:49.772 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:52:49.772 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:52:49.772 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:52:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:52:49.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:52:49.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:53:09.775 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:53:09.808 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:53:09.808 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:53:09.808 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:53:09.808 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:53:09.808 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:53:09.808 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:53:09.808 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:53:09.810 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:53:09.810 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:53:09.810 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:53:09.810 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:53:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:53:09.810 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:53:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:53:09.810 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:53:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:53:09.810 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:53:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:53:09.810 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:53:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:53:09.810 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:53:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:53:09.810 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:53:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:53:09.810 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:53:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:53:09.810 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:53:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:53:09.810 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:53:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:53:09.810 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:53:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:53:09.810 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:53:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:53:09.810 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:53:09.810 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:53:09.810 139328 ERROR oslo_service.periodic_task >2018-06-24 20:53:39.692 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:53:39.693 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:53:39.698 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:53:39.699 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:53:39.732 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:53:39.732 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:53:39.732 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:53:39.732 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:53:39.732 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:53:39.732 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:53:39.732 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:53:39.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:53:39.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:53:39.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:53:49.735 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:53:49.768 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:53:49.768 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:53:49.768 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:53:49.768 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:53:49.768 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:53:49.768 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:53:49.768 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:53:49.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:53:49.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:53:49.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:54:09.771 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:54:09.805 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:54:09.805 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:54:09.805 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:54:09.805 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:54:09.805 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:54:09.805 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:54:09.805 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:54:09.808 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:54:09.808 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:54:09.808 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:54:09.808 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:54:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:54:09.808 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:54:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:54:09.808 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:54:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:54:09.808 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:54:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:54:09.808 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:54:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:54:09.808 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:54:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:54:09.808 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:54:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:54:09.808 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:54:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:54:09.808 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:54:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:54:09.808 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:54:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:54:09.808 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:54:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:54:09.808 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:54:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:54:09.808 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:54:09.808 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:54:09.808 139328 ERROR oslo_service.periodic_task >2018-06-24 20:54:39.690 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:54:39.691 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:54:39.699 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:54:39.699 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:54:39.738 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:54:39.738 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:54:39.738 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:54:39.738 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:54:39.738 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:54:39.738 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:54:39.738 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:54:39.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:54:39.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:54:39.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:54:49.741 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:54:49.774 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:54:49.774 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:54:49.774 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:54:49.774 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:54:49.774 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:54:49.774 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:54:49.774 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:54:49.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:54:49.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:54:49.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:55:09.777 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:55:09.808 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:55:09.808 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:55:09.808 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:55:09.808 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:55:09.808 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:55:09.808 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:55:09.808 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:55:09.810 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:55:09.810 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:55:09.811 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:55:09.811 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:55:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:55:09.811 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:55:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:55:09.811 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:55:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:55:09.811 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:55:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:55:09.811 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:55:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:55:09.811 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:55:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:55:09.811 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:55:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:55:09.811 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:55:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:55:09.811 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:55:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:55:09.811 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:55:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:55:09.811 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:55:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:55:09.811 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:55:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:55:09.811 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:55:09.811 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:55:09.811 139328 ERROR oslo_service.periodic_task >2018-06-24 20:55:39.692 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:55:39.693 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:55:39.700 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:55:39.701 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:55:39.734 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:55:39.734 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:55:39.734 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:55:39.734 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:55:39.734 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:55:39.734 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:55:39.734 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:55:39.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:55:39.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:55:39.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:55:49.738 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:55:49.770 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:55:49.770 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:55:49.770 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:55:49.770 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:55:49.770 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:55:49.770 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:55:49.770 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:55:49.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:55:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:55:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:56:09.773 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:56:09.809 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:56:09.809 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:56:09.809 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:56:09.809 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:56:09.809 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:56:09.809 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:56:09.809 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:56:09.811 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:56:09.811 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:56:09.812 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:56:09.812 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:56:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:56:09.812 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:56:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:56:09.812 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:56:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:56:09.812 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:56:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:56:09.812 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:56:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:56:09.812 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:56:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:56:09.812 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:56:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:56:09.812 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:56:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:56:09.812 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:56:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:56:09.812 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:56:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:56:09.812 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:56:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:56:09.812 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:56:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:56:09.812 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:56:09.812 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:56:09.812 139328 ERROR oslo_service.periodic_task >2018-06-24 20:56:39.693 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:56:39.694 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:56:39.701 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:56:39.701 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:56:39.736 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:56:39.736 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:56:39.736 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:56:39.736 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:56:39.736 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:56:39.736 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:56:39.736 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:56:39.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:56:39.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:56:39.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:56:49.739 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:56:49.775 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:56:49.775 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:56:49.775 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:56:49.775 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:56:49.775 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:56:49.775 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:56:49.775 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:56:49.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:56:49.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:56:49.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:57:09.778 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:57:09.814 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:57:09.814 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:57:09.814 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:57:09.814 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:57:09.814 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:57:09.814 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:57:09.814 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:57:09.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:57:09.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:57:09.817 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:57:09.817 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:57:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:57:09.817 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:57:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:57:09.817 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:57:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:57:09.817 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:57:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:57:09.817 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:57:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:57:09.817 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:57:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:57:09.817 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:57:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:57:09.817 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:57:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:57:09.817 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:57:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:57:09.817 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:57:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:57:09.817 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:57:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:57:09.817 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:57:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:57:09.817 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:57:09.817 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:57:09.817 139328 ERROR oslo_service.periodic_task >2018-06-24 20:57:39.698 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:57:39.699 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:57:39.707 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:57:39.707 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:57:39.742 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:57:39.742 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:57:39.742 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:57:39.742 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:57:39.742 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:57:39.742 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:57:39.742 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:57:39.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:57:39.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:57:39.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:57:49.746 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:57:49.780 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:57:49.780 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:57:49.780 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:57:49.780 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:57:49.780 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:57:49.780 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:57:49.780 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:57:49.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:57:49.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:57:49.783 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:58:09.783 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:58:09.817 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:58:09.817 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:58:09.817 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:58:09.817 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:58:09.817 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:58:09.817 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:58:09.817 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:58:09.821 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:58:09.822 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:58:09.822 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:58:09.822 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:58:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:58:09.822 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:58:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:58:09.822 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:58:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:58:09.822 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:58:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:58:09.822 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:58:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:58:09.822 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:58:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:58:09.822 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:58:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:58:09.822 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:58:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:58:09.822 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:58:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:58:09.822 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:58:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:58:09.822 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:58:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:58:09.822 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:58:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:58:09.822 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:58:09.822 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:58:09.822 139328 ERROR oslo_service.periodic_task >2018-06-24 20:58:39.694 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:58:39.695 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:58:39.702 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:58:39.702 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:58:39.738 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:58:39.738 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:58:39.738 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:58:39.738 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:58:39.738 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:58:39.738 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:58:39.738 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:58:39.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:58:39.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:58:39.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:58:49.741 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:58:49.775 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:58:49.775 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:58:49.775 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:58:49.775 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:58:49.775 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:58:49.775 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:58:49.775 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:58:49.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:58:49.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:58:49.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:59:09.778 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:59:09.810 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:59:09.810 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:59:09.810 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:59:09.810 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:59:09.810 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:59:09.810 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:59:09.810 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:59:09.812 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:59:09.812 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:59:09.813 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:59:09.813 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 20:59:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 20:59:09.813 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 20:59:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 20:59:09.813 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 20:59:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 20:59:09.813 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 20:59:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 20:59:09.813 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 20:59:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 20:59:09.813 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 20:59:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 20:59:09.813 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 20:59:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 20:59:09.813 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 20:59:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 20:59:09.813 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 20:59:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 20:59:09.813 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 20:59:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 20:59:09.813 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 20:59:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 20:59:09.813 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 20:59:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 20:59:09.813 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 20:59:09.813 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 20:59:09.813 139328 ERROR oslo_service.periodic_task >2018-06-24 20:59:39.694 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:59:39.695 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 20:59:39.704 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 20:59:39.705 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:59:39.741 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:59:39.741 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:59:39.741 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:59:39.741 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:59:39.741 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:59:39.741 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:59:39.741 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:59:39.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:59:39.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:59:39.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 20:59:49.746 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 20:59:49.783 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 20:59:49.783 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 20:59:49.783 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 20:59:49.783 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 20:59:49.783 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 20:59:49.783 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 20:59:49.783 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 20:59:49.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 20:59:49.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.08 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 20:59:49.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:00:09.787 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:00:09.823 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:00:09.823 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:00:09.823 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:00:09.823 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:00:09.823 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:00:09.823 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:00:09.823 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:00:09.825 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:00:09.826 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.121 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:00:09.826 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:00:09.826 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:00:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:00:09.826 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:00:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:00:09.826 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:00:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:00:09.826 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:00:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:00:09.826 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:00:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:00:09.826 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:00:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:00:09.826 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:00:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:00:09.826 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:00:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:00:09.826 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:00:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:00:09.826 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:00:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:00:09.826 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:00:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:00:09.826 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:00:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:00:09.826 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:00:09.826 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:00:09.826 139328 ERROR oslo_service.periodic_task >2018-06-24 21:00:39.697 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:00:39.699 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:00:39.705 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:00:39.706 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:00:39.740 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:00:39.740 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:00:39.740 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:00:39.740 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:00:39.740 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:00:39.740 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:00:39.740 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:00:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:00:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:00:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:00:49.745 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:00:49.780 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:00:49.780 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:00:49.780 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:00:49.780 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:00:49.780 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:00:49.780 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:00:49.780 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:00:49.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:00:49.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:00:49.783 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:01:09.783 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:01:09.818 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:01:09.818 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:01:09.818 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:01:09.818 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:01:09.818 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:01:09.818 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:01:09.818 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:01:09.822 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:01:09.822 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:01:09.822 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:01:09.822 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:01:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:01:09.822 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:01:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:01:09.822 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:01:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:01:09.822 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:01:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:01:09.822 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:01:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:01:09.822 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:01:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:01:09.822 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:01:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:01:09.822 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:01:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:01:09.822 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:01:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:01:09.822 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:01:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:01:09.822 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:01:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:01:09.822 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:01:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:01:09.822 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:01:09.822 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:01:09.822 139328 ERROR oslo_service.periodic_task >2018-06-24 21:01:39.694 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:01:39.695 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:01:39.704 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:01:39.705 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:01:39.740 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:01:39.740 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:01:39.740 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:01:39.740 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:01:39.740 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:01:39.740 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:01:39.740 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:01:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:01:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:01:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:01:49.744 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:01:49.779 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:01:49.779 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:01:49.779 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:01:49.779 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:01:49.779 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:01:49.779 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:01:49.779 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:01:49.781 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:01:49.781 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:01:49.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:02:09.782 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:02:09.816 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:02:09.816 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:02:09.816 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:02:09.816 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:02:09.816 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:02:09.816 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:02:09.816 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:02:09.818 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:02:09.819 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:02:09.819 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:02:09.819 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:02:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:02:09.819 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:02:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:02:09.819 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:02:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:02:09.819 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:02:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:02:09.819 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:02:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:02:09.819 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:02:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:02:09.819 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:02:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:02:09.819 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:02:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:02:09.819 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:02:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:02:09.819 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:02:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:02:09.819 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:02:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:02:09.819 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:02:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:02:09.819 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:02:09.819 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:02:09.819 139328 ERROR oslo_service.periodic_task >2018-06-24 21:02:39.690 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:02:39.691 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:02:39.698 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:02:39.699 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:02:39.733 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:02:39.733 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:02:39.733 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:02:39.733 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:02:39.733 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:02:39.733 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:02:39.733 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:02:39.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:02:39.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:02:39.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:02:49.736 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:02:49.771 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:02:49.771 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:02:49.771 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:02:49.771 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:02:49.771 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:02:49.771 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:02:49.771 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:02:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:02:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:02:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:03:09.775 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:03:09.809 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:03:09.809 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:03:09.809 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:03:09.809 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:03:09.809 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:03:09.809 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:03:09.809 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:03:09.811 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:03:09.811 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:03:09.812 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:03:09.812 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:03:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:03:09.812 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:03:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:03:09.812 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:03:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:03:09.812 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:03:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:03:09.812 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:03:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:03:09.812 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:03:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:03:09.812 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:03:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:03:09.812 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:03:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:03:09.812 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:03:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:03:09.812 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:03:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:03:09.812 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:03:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:03:09.812 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:03:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:03:09.812 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:03:09.812 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:03:09.812 139328 ERROR oslo_service.periodic_task >2018-06-24 21:03:39.693 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:03:39.694 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:03:39.704 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:03:39.705 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:03:39.741 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:03:39.741 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:03:39.741 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:03:39.741 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:03:39.741 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:03:39.741 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:03:39.741 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:03:39.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:03:39.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:03:39.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:03:49.745 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:03:49.780 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:03:49.780 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:03:49.780 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:03:49.780 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:03:49.780 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:03:49.780 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:03:49.780 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:03:49.781 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:03:49.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:03:49.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:04:09.783 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:04:09.815 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:04:09.815 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:04:09.815 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:04:09.815 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:04:09.815 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:04:09.815 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:04:09.815 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:04:09.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:04:09.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:04:09.818 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:04:09.818 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:04:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:04:09.818 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:04:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:04:09.818 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:04:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:04:09.818 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:04:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:04:09.818 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:04:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:04:09.818 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:04:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:04:09.818 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:04:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:04:09.818 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:04:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:04:09.818 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:04:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:04:09.818 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:04:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:04:09.818 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:04:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:04:09.818 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:04:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:04:09.818 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:04:09.818 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:04:09.818 139328 ERROR oslo_service.periodic_task >2018-06-24 21:04:39.689 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:04:39.690 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:04:39.696 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:04:39.697 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:04:39.729 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:04:39.729 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:04:39.729 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:04:39.729 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:04:39.729 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:04:39.729 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:04:39.729 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:04:39.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:04:39.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:04:39.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:04:49.733 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:04:49.765 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:04:49.765 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:04:49.765 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:04:49.765 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:04:49.765 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:04:49.765 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:04:49.765 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:04:49.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:04:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:04:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:05:09.769 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:05:09.803 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:05:09.803 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:05:09.803 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:05:09.803 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:05:09.803 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:05:09.803 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:05:09.803 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:05:09.805 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:05:09.805 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:05:09.805 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:05:09.805 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:05:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:05:09.805 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:05:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:05:09.805 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:05:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:05:09.805 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:05:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:05:09.805 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:05:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:05:09.805 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:05:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:05:09.805 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:05:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:05:09.805 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:05:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:05:09.805 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:05:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:05:09.805 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:05:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:05:09.805 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:05:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:05:09.805 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:05:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:05:09.805 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:05:09.805 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:05:09.805 139328 ERROR oslo_service.periodic_task >2018-06-24 21:05:39.687 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:05:39.688 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:05:39.695 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:05:39.695 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:05:39.729 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:05:39.729 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:05:39.729 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:05:39.729 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:05:39.729 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:05:39.729 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:05:39.729 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:05:39.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:05:39.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:05:39.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:05:49.733 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:05:49.766 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:05:49.766 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:05:49.766 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:05:49.766 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:05:49.766 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:05:49.766 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:05:49.766 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:05:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:05:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:05:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:06:09.769 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:06:09.804 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:06:09.804 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:06:09.804 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:06:09.804 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:06:09.804 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:06:09.804 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:06:09.804 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:06:09.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:06:09.807 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:06:09.807 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:06:09.807 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:06:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:06:09.807 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:06:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:06:09.807 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:06:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:06:09.807 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:06:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:06:09.807 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:06:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:06:09.807 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:06:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:06:09.807 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:06:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:06:09.807 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:06:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:06:09.807 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:06:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:06:09.807 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:06:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:06:09.807 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:06:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:06:09.807 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:06:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:06:09.807 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:06:09.807 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:06:09.807 139328 ERROR oslo_service.periodic_task >2018-06-24 21:06:39.689 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:06:39.690 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:06:39.725 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:06:39.725 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:06:39.725 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:06:39.725 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:06:39.725 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:06:39.725 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:06:39.725 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:06:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:06:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:06:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:06:49.729 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:06:49.763 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:06:49.763 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:06:49.763 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:06:49.763 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:06:49.763 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:06:49.763 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:06:49.763 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:06:49.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:06:49.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:06:49.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:07:09.766 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:07:09.807 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:07:09.807 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:07:09.807 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:07:09.807 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:07:09.807 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:07:09.807 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:07:09.807 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:07:09.809 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:07:09.809 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.118 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:07:09.809 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:07:09.809 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:07:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:07:09.809 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:07:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:07:09.809 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:07:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:07:09.809 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:07:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:07:09.809 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:07:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:07:09.809 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:07:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:07:09.809 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:07:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:07:09.809 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:07:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:07:09.809 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:07:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:07:09.809 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:07:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:07:09.809 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:07:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:07:09.809 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:07:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:07:09.809 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:07:09.809 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:07:09.809 139328 ERROR oslo_service.periodic_task >2018-06-24 21:07:39.691 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:07:39.692 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:07:39.698 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:07:39.699 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:07:39.739 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:07:39.739 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:07:39.739 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:07:39.739 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:07:39.739 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:07:39.739 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:07:39.739 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:07:39.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:07:39.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.042 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:07:39.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:07:49.742 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:07:49.781 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:07:49.781 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:07:49.781 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:07:49.781 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:07:49.781 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:07:49.781 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:07:49.781 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:07:49.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:07:49.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.085 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:07:49.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:08:09.785 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:08:09.823 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:08:09.823 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:08:09.823 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:08:09.823 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:08:09.823 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:08:09.823 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:08:09.823 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:08:09.826 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:08:09.826 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.127 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:08:09.827 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:08:09.827 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:08:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:08:09.827 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:08:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:08:09.827 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:08:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:08:09.827 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:08:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:08:09.827 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:08:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:08:09.827 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:08:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:08:09.827 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:08:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:08:09.827 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:08:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:08:09.827 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:08:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:08:09.827 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:08:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:08:09.827 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:08:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:08:09.827 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:08:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:08:09.827 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:08:09.827 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:08:09.827 139328 ERROR oslo_service.periodic_task >2018-06-24 21:08:39.689 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:08:39.690 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:08:39.698 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:08:39.698 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:08:39.732 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:08:39.732 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:08:39.732 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:08:39.732 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:08:39.732 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:08:39.732 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:08:39.732 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:08:39.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:08:39.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:08:39.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:08:49.735 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:08:49.773 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:08:49.773 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:08:49.773 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:08:49.773 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:08:49.773 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:08:49.773 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:08:49.773 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:08:49.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:08:49.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:08:49.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:09:09.777 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:09:09.811 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:09:09.811 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:09:09.811 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:09:09.811 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:09:09.811 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:09:09.811 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:09:09.811 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:09:09.813 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:09:09.813 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:09:09.814 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:09:09.814 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:09:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:09:09.814 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:09:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:09:09.814 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:09:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:09:09.814 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:09:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:09:09.814 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:09:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:09:09.814 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:09:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:09:09.814 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:09:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:09:09.814 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:09:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:09:09.814 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:09:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:09:09.814 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:09:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:09:09.814 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:09:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:09:09.814 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:09:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:09:09.814 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:09:09.814 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:09:09.814 139328 ERROR oslo_service.periodic_task >2018-06-24 21:09:39.685 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:09:39.686 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:09:39.693 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:09:39.694 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:09:39.728 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:09:39.728 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:09:39.728 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:09:39.728 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:09:39.728 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:09:39.728 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:09:39.728 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:09:39.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:09:39.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:09:39.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:09:49.731 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:09:49.767 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:09:49.767 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:09:49.767 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:09:49.767 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:09:49.767 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:09:49.767 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:09:49.767 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:09:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:09:49.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:09:49.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:10:09.771 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:10:09.805 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:10:09.805 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:10:09.805 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:10:09.805 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:10:09.805 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:10:09.805 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:10:09.805 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:10:09.807 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:10:09.807 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:10:09.807 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:10:09.807 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:10:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:10:09.807 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:10:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:10:09.807 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:10:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:10:09.807 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:10:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:10:09.807 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:10:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:10:09.807 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:10:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:10:09.807 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:10:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:10:09.807 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:10:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:10:09.807 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:10:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:10:09.807 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:10:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:10:09.807 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:10:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:10:09.807 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:10:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:10:09.807 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:10:09.807 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:10:09.807 139328 ERROR oslo_service.periodic_task >2018-06-24 21:10:39.680 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:10:39.682 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:10:39.689 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:10:39.690 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:10:39.725 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:10:39.725 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:10:39.725 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:10:39.725 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:10:39.725 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:10:39.725 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:10:39.725 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:10:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:10:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:10:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:10:49.729 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:10:49.762 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:10:49.762 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:10:49.762 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:10:49.762 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:10:49.762 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:10:49.762 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:10:49.762 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:10:49.764 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:10:49.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:10:49.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:11:09.765 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:11:09.799 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:11:09.799 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:11:09.799 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:11:09.799 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:11:09.799 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:11:09.799 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:11:09.799 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:11:09.801 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:11:09.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:11:09.802 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:11:09.802 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:11:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:11:09.802 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:11:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:11:09.802 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:11:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:11:09.802 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:11:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:11:09.802 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:11:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:11:09.802 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:11:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:11:09.802 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:11:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:11:09.802 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:11:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:11:09.802 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:11:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:11:09.802 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:11:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:11:09.802 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:11:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:11:09.802 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:11:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:11:09.802 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:11:09.802 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:11:09.802 139328 ERROR oslo_service.periodic_task >2018-06-24 21:11:39.684 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:11:39.685 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:11:39.692 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:11:39.693 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:11:39.726 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:11:39.726 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:11:39.726 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:11:39.726 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:11:39.726 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:11:39.726 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:11:39.726 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:11:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:11:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:11:39.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:11:49.729 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:11:49.765 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:11:49.765 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:11:49.765 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:11:49.765 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:11:49.765 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:11:49.765 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:11:49.765 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:11:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:11:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:11:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:12:09.769 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:12:09.804 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:12:09.804 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:12:09.804 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:12:09.804 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:12:09.804 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:12:09.804 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:12:09.804 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:12:09.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:12:09.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:12:09.807 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:12:09.807 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:12:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:12:09.807 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:12:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:12:09.807 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:12:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:12:09.807 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:12:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:12:09.807 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:12:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:12:09.807 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:12:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:12:09.807 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:12:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:12:09.807 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:12:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:12:09.807 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:12:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:12:09.807 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:12:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:12:09.807 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:12:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:12:09.807 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:12:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:12:09.807 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:12:09.807 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:12:09.807 139328 ERROR oslo_service.periodic_task >2018-06-24 21:12:39.689 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:12:39.690 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:12:39.696 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:12:39.697 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:12:39.731 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:12:39.731 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:12:39.731 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:12:39.731 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:12:39.731 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:12:39.731 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:12:39.731 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:12:39.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:12:39.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:12:39.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:12:49.734 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:12:49.767 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:12:49.767 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:12:49.767 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:12:49.767 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:12:49.767 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:12:49.767 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:12:49.767 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:12:49.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:12:49.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:12:49.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:13:09.771 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:13:09.805 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:13:09.805 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:13:09.805 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:13:09.805 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:13:09.805 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:13:09.805 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:13:09.805 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:13:09.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:13:09.807 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:13:09.807 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:13:09.807 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:13:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:13:09.807 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:13:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:13:09.807 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:13:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:13:09.807 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:13:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:13:09.807 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:13:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:13:09.807 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:13:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:13:09.807 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:13:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:13:09.807 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:13:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:13:09.807 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:13:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:13:09.807 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:13:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:13:09.807 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:13:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:13:09.807 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:13:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:13:09.807 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:13:09.807 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:13:09.807 139328 ERROR oslo_service.periodic_task >2018-06-24 21:13:39.688 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:13:39.689 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:13:39.697 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:13:39.698 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:13:39.733 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:13:39.733 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:13:39.733 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:13:39.733 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:13:39.733 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:13:39.733 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:13:39.733 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:13:39.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:13:39.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:13:39.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:13:49.736 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:13:49.771 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:13:49.771 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:13:49.771 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:13:49.771 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:13:49.771 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:13:49.771 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:13:49.771 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:13:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:13:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:13:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:14:09.774 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:14:09.819 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:14:09.819 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:14:09.819 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:14:09.819 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:14:09.819 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:14:09.819 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:14:09.819 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:14:09.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:14:09.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.125 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:14:09.824 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:14:09.824 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:14:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:14:09.824 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:14:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:14:09.824 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:14:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:14:09.824 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:14:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:14:09.824 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:14:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:14:09.824 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:14:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:14:09.824 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:14:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:14:09.824 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:14:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:14:09.824 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:14:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:14:09.824 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:14:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:14:09.824 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:14:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:14:09.824 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:14:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:14:09.824 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:14:09.824 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:14:09.824 139328 ERROR oslo_service.periodic_task >2018-06-24 21:14:39.686 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:14:39.687 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:14:39.693 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:14:39.694 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:14:39.727 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:14:39.727 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:14:39.727 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:14:39.727 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:14:39.727 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:14:39.727 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:14:39.727 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:14:39.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:14:39.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:14:39.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:14:49.731 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:14:49.768 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:14:49.768 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:14:49.768 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:14:49.768 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:14:49.768 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:14:49.768 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:14:49.768 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:14:49.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:14:49.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:14:49.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:15:09.772 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:15:09.808 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:15:09.808 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:15:09.808 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:15:09.808 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:15:09.808 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:15:09.808 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:15:09.808 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:15:09.810 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:15:09.810 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:15:09.810 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:15:09.810 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:15:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:15:09.810 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:15:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:15:09.810 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:15:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:15:09.810 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:15:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:15:09.810 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:15:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:15:09.810 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:15:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:15:09.810 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:15:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:15:09.810 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:15:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:15:09.810 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:15:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:15:09.810 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:15:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:15:09.810 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:15:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:15:09.810 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:15:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:15:09.810 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:15:09.810 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:15:09.810 139328 ERROR oslo_service.periodic_task >2018-06-24 21:15:39.682 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:15:39.683 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:15:39.692 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:15:39.692 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:15:39.730 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:15:39.730 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:15:39.730 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:15:39.730 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:15:39.730 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:15:39.730 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:15:39.730 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:15:39.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:15:39.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:15:39.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:15:49.734 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:15:49.766 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:15:49.766 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:15:49.766 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:15:49.766 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:15:49.766 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:15:49.766 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:15:49.766 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:15:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:15:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:15:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:16:09.769 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:16:09.803 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:16:09.803 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:16:09.803 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:16:09.803 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:16:09.803 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:16:09.803 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:16:09.803 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:16:09.805 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:16:09.805 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:16:09.805 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:16:09.805 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:16:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:16:09.805 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:16:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:16:09.805 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:16:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:16:09.805 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:16:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:16:09.805 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:16:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:16:09.805 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:16:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:16:09.805 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:16:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:16:09.805 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:16:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:16:09.805 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:16:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:16:09.805 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:16:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:16:09.805 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:16:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:16:09.805 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:16:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:16:09.805 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:16:09.805 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:16:09.805 139328 ERROR oslo_service.periodic_task >2018-06-24 21:16:39.687 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:16:39.688 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:16:39.695 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:16:39.695 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:16:39.728 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:16:39.728 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:16:39.728 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:16:39.728 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:16:39.728 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:16:39.728 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:16:39.728 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:16:39.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:16:39.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:16:39.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:16:49.732 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:16:49.765 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:16:49.765 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:16:49.765 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:16:49.765 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:16:49.765 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:16:49.765 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:16:49.765 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:16:49.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:16:49.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:16:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:17:09.768 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:17:09.801 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:17:09.801 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:17:09.801 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:17:09.801 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:17:09.801 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:17:09.801 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:17:09.801 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:17:09.803 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:17:09.804 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:17:09.804 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:17:09.804 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:17:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:17:09.804 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:17:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:17:09.804 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:17:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:17:09.804 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:17:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:17:09.804 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:17:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:17:09.804 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:17:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:17:09.804 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:17:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:17:09.804 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:17:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:17:09.804 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:17:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:17:09.804 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:17:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:17:09.804 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:17:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:17:09.804 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:17:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:17:09.804 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:17:09.804 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:17:09.804 139328 ERROR oslo_service.periodic_task >2018-06-24 21:17:39.686 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:17:39.687 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:17:39.693 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:17:39.694 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:17:39.727 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:17:39.727 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:17:39.727 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:17:39.727 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:17:39.727 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:17:39.727 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:17:39.727 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:17:39.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:17:39.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:17:39.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:17:49.731 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:17:49.763 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:17:49.763 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:17:49.763 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:17:49.763 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:17:49.763 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:17:49.763 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:17:49.763 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:17:49.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:17:49.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:17:49.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:18:09.766 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:18:09.801 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:18:09.801 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:18:09.801 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:18:09.801 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:18:09.801 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:18:09.801 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:18:09.801 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:18:09.803 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:18:09.804 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:18:09.804 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:18:09.804 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:18:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:18:09.804 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:18:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:18:09.804 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:18:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:18:09.804 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:18:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:18:09.804 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:18:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:18:09.804 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:18:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:18:09.804 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:18:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:18:09.804 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:18:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:18:09.804 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:18:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:18:09.804 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:18:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:18:09.804 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:18:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:18:09.804 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:18:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:18:09.804 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:18:09.804 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:18:09.804 139328 ERROR oslo_service.periodic_task >2018-06-24 21:18:39.686 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:18:39.687 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:18:39.695 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:18:39.695 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:18:39.730 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:18:39.730 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:18:39.730 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:18:39.730 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:18:39.730 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:18:39.730 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:18:39.730 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:18:39.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:18:39.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:18:39.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:18:49.733 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:18:49.770 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:18:49.770 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:18:49.770 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:18:49.770 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:18:49.770 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:18:49.770 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:18:49.770 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:18:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:18:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:18:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:19:09.774 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:19:09.807 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:19:09.807 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:19:09.807 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:19:09.807 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:19:09.807 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:19:09.807 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:19:09.807 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:19:09.809 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:19:09.810 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:19:09.810 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:19:09.810 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:19:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:19:09.810 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:19:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:19:09.810 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:19:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:19:09.810 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:19:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:19:09.810 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:19:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:19:09.810 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:19:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:19:09.810 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:19:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:19:09.810 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:19:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:19:09.810 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:19:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:19:09.810 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:19:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:19:09.810 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:19:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:19:09.810 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:19:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:19:09.810 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:19:09.810 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:19:09.810 139328 ERROR oslo_service.periodic_task >2018-06-24 21:19:39.681 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:19:39.683 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:19:39.690 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:19:39.690 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:19:39.725 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:19:39.725 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:19:39.725 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:19:39.725 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:19:39.725 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:19:39.725 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:19:39.725 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:19:39.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:19:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:19:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:19:49.728 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:19:49.763 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:19:49.763 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:19:49.763 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:19:49.763 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:19:49.763 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:19:49.763 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:19:49.763 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:19:49.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:19:49.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:19:49.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:20:09.766 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:20:09.800 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:20:09.800 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:20:09.800 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:20:09.800 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:20:09.800 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:20:09.800 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:20:09.800 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:20:09.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:20:09.803 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:20:09.803 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:20:09.803 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:20:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:20:09.803 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:20:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:20:09.803 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:20:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:20:09.803 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:20:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:20:09.803 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:20:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:20:09.803 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:20:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:20:09.803 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:20:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:20:09.803 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:20:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:20:09.803 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:20:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:20:09.803 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:20:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:20:09.803 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:20:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:20:09.803 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:20:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:20:09.803 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:20:09.803 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:20:09.803 139328 ERROR oslo_service.periodic_task >2018-06-24 21:20:39.684 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:20:39.685 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:20:39.692 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:20:39.693 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:20:39.728 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:20:39.728 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:20:39.728 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:20:39.728 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:20:39.728 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:20:39.728 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:20:39.728 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:20:39.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:20:39.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:20:39.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:20:49.732 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:20:49.771 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:20:49.771 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:20:49.771 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:20:49.771 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:20:49.771 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:20:49.771 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:20:49.771 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:20:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:20:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.082 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:20:49.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:21:09.776 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:21:09.810 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:21:09.810 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:21:09.810 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:21:09.810 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:21:09.810 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:21:09.810 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:21:09.810 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:21:09.812 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:21:09.812 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.12 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:21:09.813 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:21:09.813 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:21:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:21:09.813 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:21:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:21:09.813 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:21:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:21:09.813 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:21:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:21:09.813 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:21:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:21:09.813 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:21:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:21:09.813 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:21:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:21:09.813 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:21:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:21:09.813 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:21:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:21:09.813 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:21:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:21:09.813 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:21:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:21:09.813 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:21:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:21:09.813 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:21:09.813 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:21:09.813 139328 ERROR oslo_service.periodic_task >2018-06-24 21:21:39.684 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:21:39.685 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:21:39.692 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:21:39.693 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:21:39.727 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:21:39.727 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:21:39.727 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:21:39.727 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:21:39.727 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:21:39.727 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:21:39.727 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:21:39.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:21:39.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:21:39.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:21:49.731 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:21:49.763 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:21:49.763 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:21:49.763 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:21:49.763 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:21:49.763 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:21:49.763 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:21:49.763 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:21:49.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:21:49.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:21:49.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:22:09.766 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:22:09.802 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:22:09.802 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:22:09.802 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:22:09.802 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:22:09.802 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:22:09.802 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:22:09.802 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:22:09.804 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:22:09.804 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:22:09.805 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:22:09.805 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:22:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:22:09.805 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:22:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:22:09.805 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:22:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:22:09.805 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:22:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:22:09.805 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:22:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:22:09.805 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:22:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:22:09.805 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:22:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:22:09.805 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:22:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:22:09.805 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:22:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:22:09.805 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:22:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:22:09.805 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:22:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:22:09.805 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:22:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:22:09.805 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:22:09.805 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:22:09.805 139328 ERROR oslo_service.periodic_task >2018-06-24 21:22:39.686 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:22:39.687 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:22:39.695 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:22:39.696 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:22:39.747 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:22:39.747 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:22:39.747 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:22:39.747 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:22:39.747 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:22:39.747 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:22:39.747 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:22:39.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:22:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.053 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:22:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:22:49.751 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:22:49.783 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:22:49.783 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:22:49.783 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:22:49.783 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:22:49.783 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:22:49.783 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:22:49.783 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:22:49.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:22:49.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.089 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:22:49.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:23:09.786 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:23:09.820 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:23:09.820 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:23:09.820 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:23:09.820 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:23:09.820 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:23:09.820 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:23:09.820 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:23:09.822 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:23:09.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.126 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:23:09.823 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:23:09.823 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:23:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:23:09.823 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:23:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:23:09.823 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:23:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:23:09.823 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:23:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:23:09.823 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:23:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:23:09.823 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:23:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:23:09.823 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:23:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:23:09.823 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:23:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:23:09.823 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:23:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:23:09.823 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:23:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:23:09.823 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:23:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:23:09.823 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:23:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:23:09.823 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:23:09.823 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:23:09.823 139328 ERROR oslo_service.periodic_task >2018-06-24 21:23:39.684 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:23:39.686 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:23:39.692 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:23:39.693 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:23:39.726 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:23:39.726 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:23:39.726 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:23:39.726 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:23:39.726 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:23:39.726 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:23:39.726 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:23:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:23:39.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:23:39.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:23:49.729 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:23:49.766 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:23:49.766 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:23:49.766 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:23:49.766 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:23:49.766 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:23:49.766 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:23:49.766 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:23:49.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:23:49.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:23:49.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:24:09.772 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:24:09.803 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:24:09.803 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:24:09.803 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:24:09.803 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:24:09.803 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:24:09.803 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:24:09.803 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:24:09.805 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:24:09.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:24:09.806 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:24:09.806 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:24:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:24:09.806 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:24:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:24:09.806 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:24:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:24:09.806 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:24:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:24:09.806 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:24:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:24:09.806 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:24:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:24:09.806 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:24:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:24:09.806 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:24:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:24:09.806 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:24:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:24:09.806 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:24:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:24:09.806 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:24:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:24:09.806 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:24:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:24:09.806 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:24:09.806 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:24:09.806 139328 ERROR oslo_service.periodic_task >2018-06-24 21:24:39.688 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:24:39.688 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:24:39.696 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:24:39.696 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:24:39.732 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:24:39.732 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:24:39.732 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:24:39.732 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:24:39.732 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:24:39.732 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:24:39.732 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:24:39.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:24:39.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:24:39.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:24:49.735 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:24:49.770 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:24:49.770 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:24:49.770 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:24:49.770 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:24:49.770 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:24:49.770 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:24:49.770 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:24:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:24:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:24:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:25:09.774 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:25:09.812 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:25:09.812 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:25:09.812 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:25:09.812 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:25:09.812 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:25:09.812 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:25:09.812 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:25:09.814 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:25:09.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.118 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:25:09.815 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:25:09.815 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:25:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:25:09.815 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:25:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:25:09.815 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:25:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:25:09.815 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:25:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:25:09.815 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:25:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:25:09.815 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:25:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:25:09.815 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:25:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:25:09.815 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:25:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:25:09.815 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:25:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:25:09.815 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:25:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:25:09.815 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:25:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:25:09.815 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:25:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:25:09.815 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:25:09.815 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:25:09.815 139328 ERROR oslo_service.periodic_task >2018-06-24 21:25:39.687 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:25:39.688 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:25:39.696 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:25:39.697 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:25:39.730 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:25:39.730 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:25:39.730 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:25:39.730 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:25:39.730 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:25:39.730 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:25:39.730 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:25:39.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:25:39.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:25:39.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:25:49.733 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:25:49.769 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:25:49.769 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:25:49.769 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:25:49.769 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:25:49.769 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:25:49.769 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:25:49.769 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:25:49.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:25:49.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:25:49.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:26:09.773 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:26:09.807 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:26:09.807 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:26:09.807 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:26:09.807 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:26:09.807 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:26:09.807 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:26:09.807 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:26:09.810 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:26:09.810 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:26:09.810 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:26:09.810 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:26:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:26:09.810 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:26:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:26:09.810 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:26:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:26:09.810 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:26:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:26:09.810 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:26:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:26:09.810 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:26:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:26:09.810 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:26:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:26:09.810 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:26:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:26:09.810 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:26:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:26:09.810 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:26:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:26:09.810 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:26:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:26:09.810 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:26:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:26:09.810 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:26:09.810 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:26:09.810 139328 ERROR oslo_service.periodic_task >2018-06-24 21:26:39.682 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:26:39.683 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:26:39.690 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:26:39.691 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:26:39.726 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:26:39.726 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:26:39.726 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:26:39.726 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:26:39.726 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:26:39.726 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:26:39.726 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:26:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:26:39.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:26:39.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:26:49.730 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:26:49.764 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:26:49.764 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:26:49.764 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:26:49.764 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:26:49.764 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:26:49.764 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:26:49.764 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:26:49.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:26:49.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:26:49.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:27:09.767 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:27:09.801 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:27:09.801 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:27:09.801 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:27:09.801 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:27:09.801 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:27:09.801 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:27:09.801 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:27:09.803 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:27:09.803 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:27:09.803 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:27:09.803 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:27:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:27:09.803 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:27:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:27:09.803 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:27:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:27:09.803 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:27:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:27:09.803 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:27:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:27:09.803 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:27:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:27:09.803 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:27:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:27:09.803 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:27:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:27:09.803 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:27:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:27:09.803 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:27:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:27:09.803 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:27:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:27:09.803 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:27:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:27:09.803 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:27:09.803 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:27:09.803 139328 ERROR oslo_service.periodic_task >2018-06-24 21:27:39.685 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:27:39.686 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:27:39.692 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:27:39.693 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:27:39.725 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:27:39.725 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:27:39.725 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:27:39.725 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:27:39.725 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:27:39.725 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:27:39.725 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:27:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:27:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:27:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:27:49.729 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:27:49.767 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:27:49.767 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:27:49.767 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:27:49.767 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:27:49.767 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:27:49.767 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:27:49.767 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:27:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:27:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:27:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:28:09.770 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:28:09.803 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:28:09.803 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:28:09.803 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:28:09.803 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:28:09.803 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:28:09.803 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:28:09.803 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:28:09.805 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:28:09.805 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:28:09.806 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:28:09.806 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:28:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:28:09.806 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:28:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:28:09.806 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:28:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:28:09.806 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:28:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:28:09.806 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:28:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:28:09.806 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:28:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:28:09.806 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:28:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:28:09.806 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:28:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:28:09.806 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:28:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:28:09.806 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:28:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:28:09.806 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:28:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:28:09.806 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:28:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:28:09.806 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:28:09.806 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:28:09.806 139328 ERROR oslo_service.periodic_task >2018-06-24 21:28:39.688 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:28:39.689 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:28:39.697 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:28:39.698 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:28:39.735 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:28:39.735 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:28:39.735 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:28:39.735 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:28:39.735 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:28:39.735 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:28:39.735 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:28:39.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:28:39.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:28:39.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:28:49.738 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:28:49.771 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:28:49.771 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:28:49.771 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:28:49.771 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:28:49.771 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:28:49.771 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:28:49.771 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:28:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:28:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:28:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:29:09.775 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:29:09.808 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:29:09.808 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:29:09.808 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:29:09.808 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:29:09.808 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:29:09.808 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:29:09.808 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:29:09.810 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:29:09.811 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:29:09.811 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:29:09.811 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:29:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:29:09.811 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:29:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:29:09.811 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:29:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:29:09.811 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:29:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:29:09.811 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:29:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:29:09.811 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:29:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:29:09.811 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:29:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:29:09.811 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:29:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:29:09.811 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:29:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:29:09.811 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:29:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:29:09.811 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:29:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:29:09.811 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:29:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:29:09.811 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:29:09.811 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:29:09.811 139328 ERROR oslo_service.periodic_task >2018-06-24 21:29:39.692 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:29:39.693 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:29:39.700 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:29:39.700 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:29:39.734 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:29:39.734 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:29:39.734 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:29:39.734 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:29:39.734 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:29:39.734 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:29:39.734 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:29:39.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:29:39.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:29:39.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:29:49.738 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:29:49.771 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:29:49.771 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:29:49.771 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:29:49.771 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:29:49.771 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:29:49.771 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:29:49.771 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:29:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:29:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:29:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:30:09.774 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:30:09.809 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:30:09.809 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:30:09.809 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:30:09.809 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:30:09.809 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:30:09.809 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:30:09.809 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:30:09.811 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:30:09.811 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:30:09.811 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:30:09.811 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:30:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:30:09.811 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:30:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:30:09.811 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:30:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:30:09.811 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:30:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:30:09.811 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:30:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:30:09.811 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:30:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:30:09.811 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:30:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:30:09.811 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:30:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:30:09.811 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:30:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:30:09.811 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:30:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:30:09.811 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:30:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:30:09.811 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:30:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:30:09.811 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:30:09.811 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:30:09.811 139328 ERROR oslo_service.periodic_task >2018-06-24 21:30:39.693 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:30:39.694 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:30:39.707 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:30:39.707 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:30:39.744 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:30:39.744 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:30:39.744 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:30:39.744 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:30:39.744 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:30:39.744 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:30:39.744 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:30:39.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:30:39.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:30:39.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:30:49.748 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:30:49.780 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:30:49.780 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:30:49.780 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:30:49.780 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:30:49.780 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:30:49.780 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:30:49.780 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:30:49.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:30:49.783 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:30:49.783 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:31:09.783 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:31:09.816 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:31:09.816 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:31:09.816 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:31:09.816 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:31:09.816 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:31:09.816 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:31:09.816 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:31:09.818 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:31:09.818 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:31:09.818 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:31:09.818 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:31:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:31:09.818 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:31:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:31:09.818 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:31:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:31:09.818 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:31:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:31:09.818 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:31:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:31:09.818 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:31:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:31:09.818 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:31:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:31:09.818 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:31:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:31:09.818 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:31:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:31:09.818 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:31:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:31:09.818 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:31:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:31:09.818 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:31:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:31:09.818 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:31:09.818 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:31:09.818 139328 ERROR oslo_service.periodic_task >2018-06-24 21:31:39.690 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:31:39.691 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:31:39.701 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:31:39.702 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:31:39.735 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:31:39.735 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:31:39.735 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:31:39.735 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:31:39.735 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:31:39.735 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:31:39.735 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:31:39.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:31:39.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:31:39.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:31:49.738 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:31:49.771 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:31:49.771 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:31:49.771 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:31:49.771 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:31:49.771 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:31:49.771 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:31:49.771 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:31:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:31:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:31:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:32:09.774 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:32:09.808 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:32:09.808 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:32:09.808 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:32:09.808 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:32:09.808 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:32:09.808 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:32:09.808 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:32:09.809 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:32:09.810 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:32:09.810 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:32:09.810 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:32:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:32:09.810 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:32:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:32:09.810 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:32:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:32:09.810 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:32:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:32:09.810 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:32:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:32:09.810 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:32:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:32:09.810 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:32:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:32:09.810 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:32:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:32:09.810 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:32:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:32:09.810 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:32:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:32:09.810 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:32:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:32:09.810 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:32:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:32:09.810 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:32:09.810 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:32:09.810 139328 ERROR oslo_service.periodic_task >2018-06-24 21:32:39.692 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:32:39.693 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:32:39.701 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:32:39.702 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:32:39.739 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:32:39.739 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:32:39.739 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:32:39.739 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:32:39.739 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:32:39.739 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:32:39.739 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:32:39.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:32:39.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:32:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:32:49.743 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:32:49.777 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:32:49.777 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:32:49.777 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:32:49.777 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:32:49.777 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:32:49.777 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:32:49.777 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:32:49.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:32:49.781 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:32:49.781 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:33:09.781 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:33:09.815 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:33:09.815 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:33:09.815 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:33:09.815 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:33:09.815 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:33:09.815 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:33:09.815 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:33:09.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:33:09.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:33:09.817 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:33:09.817 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:33:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:33:09.817 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:33:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:33:09.817 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:33:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:33:09.817 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:33:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:33:09.817 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:33:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:33:09.817 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:33:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:33:09.817 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:33:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:33:09.817 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:33:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:33:09.817 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:33:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:33:09.817 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:33:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:33:09.817 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:33:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:33:09.817 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:33:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:33:09.817 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:33:09.817 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:33:09.817 139328 ERROR oslo_service.periodic_task >2018-06-24 21:33:39.689 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:33:39.690 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:33:39.696 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:33:39.697 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:33:39.734 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:33:39.734 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:33:39.734 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:33:39.734 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:33:39.734 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:33:39.734 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:33:39.734 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:33:39.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:33:39.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:33:39.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:33:49.738 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:33:49.782 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:33:49.782 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:33:49.782 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:33:49.782 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:33:49.782 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:33:49.782 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:33:49.782 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:33:49.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:33:49.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.088 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:33:49.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:34:09.786 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:34:09.834 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:34:09.834 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:34:09.834 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:34:09.834 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:34:09.834 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:34:09.834 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:34:09.834 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:34:09.836 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:34:09.837 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.14 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:34:09.837 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:34:09.837 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:34:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:34:09.837 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:34:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:34:09.837 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:34:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:34:09.837 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:34:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:34:09.837 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:34:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:34:09.837 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:34:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:34:09.837 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:34:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:34:09.837 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:34:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:34:09.837 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:34:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:34:09.837 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:34:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:34:09.837 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:34:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:34:09.837 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:34:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:34:09.837 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:34:09.837 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:34:09.837 139328 ERROR oslo_service.periodic_task >2018-06-24 21:34:39.688 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:34:39.690 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:34:39.698 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:34:39.699 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:34:39.732 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:34:39.732 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:34:39.732 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:34:39.732 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:34:39.732 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:34:39.732 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:34:39.732 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:34:39.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:34:39.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:34:39.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:34:49.736 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:34:49.769 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:34:49.769 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:34:49.769 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:34:49.769 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:34:49.769 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:34:49.769 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:34:49.769 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:34:49.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:34:49.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:34:49.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:35:09.773 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:35:09.808 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:35:09.808 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:35:09.808 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:35:09.808 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:35:09.808 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:35:09.808 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:35:09.808 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:35:09.810 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:35:09.811 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:35:09.811 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:35:09.811 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:35:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:35:09.811 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:35:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:35:09.811 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:35:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:35:09.811 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:35:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:35:09.811 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:35:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:35:09.811 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:35:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:35:09.811 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:35:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:35:09.811 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:35:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:35:09.811 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:35:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:35:09.811 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:35:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:35:09.811 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:35:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:35:09.811 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:35:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:35:09.811 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:35:09.811 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:35:09.811 139328 ERROR oslo_service.periodic_task >2018-06-24 21:35:39.692 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:35:39.693 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:36:39.690 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:36:39.692 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:36:39.698 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:36:39.698 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:36:39.735 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:36:39.735 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:36:39.735 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:36:39.735 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:36:39.735 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:36:39.735 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:36:39.735 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:36:39.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:36:39.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:36:39.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:36:49.740 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:36:49.774 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:36:49.774 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:36:49.774 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:36:49.774 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:36:49.774 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:36:49.774 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:36:49.774 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:36:49.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:36:49.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:36:49.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:37:09.778 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:37:09.821 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:37:09.821 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:37:09.821 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:37:09.821 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:37:09.821 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:37:09.821 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:37:09.821 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:37:09.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:37:09.824 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.125 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:37:09.824 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:37:09.824 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:37:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:37:09.824 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:37:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:37:09.824 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:37:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:37:09.824 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:37:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:37:09.824 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:37:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:37:09.824 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:37:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:37:09.824 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:37:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:37:09.824 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:37:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:37:09.824 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:37:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:37:09.824 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:37:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:37:09.824 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:37:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:37:09.824 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:37:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:37:09.824 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:37:09.824 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:37:09.824 139328 ERROR oslo_service.periodic_task >2018-06-24 21:37:39.685 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:37:39.687 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:37:39.693 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:37:39.693 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:37:39.728 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:37:39.728 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:37:39.728 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:37:39.728 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:37:39.728 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:37:39.728 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:37:39.728 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:37:39.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:37:39.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:37:39.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:37:49.732 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:37:49.765 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:37:49.765 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:37:49.765 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:37:49.765 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:37:49.765 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:37:49.765 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:37:49.765 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:37:49.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:37:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:37:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:38:09.768 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:38:09.808 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:38:09.808 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:38:09.808 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:38:09.808 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:38:09.808 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:38:09.808 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:38:09.808 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:38:09.810 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:38:09.810 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:38:09.810 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:38:09.810 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:38:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:38:09.810 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:38:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:38:09.810 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:38:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:38:09.810 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:38:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:38:09.810 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:38:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:38:09.810 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:38:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:38:09.810 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:38:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:38:09.810 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:38:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:38:09.810 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:38:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:38:09.810 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:38:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:38:09.810 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:38:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:38:09.810 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:38:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:38:09.810 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:38:09.810 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:38:09.810 139328 ERROR oslo_service.periodic_task >2018-06-24 21:38:39.682 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:38:39.683 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:38:39.691 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:38:39.691 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:38:39.726 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:38:39.726 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:38:39.726 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:38:39.726 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:38:39.726 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:38:39.726 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:38:39.726 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:38:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:38:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:38:39.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:38:49.729 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:38:49.762 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:38:49.762 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:38:49.762 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:38:49.762 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:38:49.762 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:38:49.762 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:38:49.762 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:38:49.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:38:49.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:38:49.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:39:09.766 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:39:09.803 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:39:09.803 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:39:09.803 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:39:09.803 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:39:09.803 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:39:09.803 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:39:09.803 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:39:09.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:39:09.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:39:09.807 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:39:09.807 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:39:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:39:09.807 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:39:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:39:09.807 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:39:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:39:09.807 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:39:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:39:09.807 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:39:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:39:09.807 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:39:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:39:09.807 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:39:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:39:09.807 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:39:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:39:09.807 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:39:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:39:09.807 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:39:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:39:09.807 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:39:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:39:09.807 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:39:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:39:09.807 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:39:09.807 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:39:09.807 139328 ERROR oslo_service.periodic_task >2018-06-24 21:39:39.679 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:39:39.680 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:39:39.696 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:39:39.697 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:39:39.732 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:39:39.732 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:39:39.732 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:39:39.732 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:39:39.732 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:39:39.732 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:39:39.732 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:39:39.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:39:39.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:39:39.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:39:49.736 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:39:49.768 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:39:49.768 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:39:49.768 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:39:49.768 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:39:49.768 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:39:49.768 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:39:49.768 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:39:49.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:39:49.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:39:49.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:40:09.772 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:40:09.805 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:40:09.805 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:40:09.805 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:40:09.805 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:40:09.805 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:40:09.805 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:40:09.805 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:40:09.807 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:40:09.807 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:40:09.808 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:40:09.808 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:40:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:40:09.808 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:40:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:40:09.808 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:40:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:40:09.808 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:40:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:40:09.808 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:40:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:40:09.808 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:40:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:40:09.808 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:40:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:40:09.808 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:40:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:40:09.808 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:40:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:40:09.808 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:40:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:40:09.808 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:40:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:40:09.808 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:40:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:40:09.808 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:40:09.808 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:40:09.808 139328 ERROR oslo_service.periodic_task >2018-06-24 21:40:39.679 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:40:39.680 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:40:39.687 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:40:39.688 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:40:39.722 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:40:39.722 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:40:39.722 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:40:39.722 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:40:39.722 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:40:39.722 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:40:39.722 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:40:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:40:39.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:40:39.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:40:49.725 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:40:49.760 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:40:49.760 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:40:49.760 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:40:49.760 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:40:49.760 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:40:49.760 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:40:49.760 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:40:49.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:40:49.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:40:49.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:41:09.764 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:41:09.798 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:41:09.798 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:41:09.798 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:41:09.798 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:41:09.798 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:41:09.798 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:41:09.798 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:41:09.801 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:41:09.801 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:41:09.801 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:41:09.801 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:41:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:41:09.801 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:41:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:41:09.801 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:41:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:41:09.801 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:41:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:41:09.801 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:41:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:41:09.801 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:41:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:41:09.801 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:41:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:41:09.801 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:41:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:41:09.801 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:41:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:41:09.801 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:41:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:41:09.801 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:41:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:41:09.801 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:41:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:41:09.801 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:41:09.801 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:41:09.801 139328 ERROR oslo_service.periodic_task >2018-06-24 21:41:39.683 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:41:39.684 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:41:39.691 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:41:39.692 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:41:39.729 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:41:39.729 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:41:39.729 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:41:39.729 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:41:39.729 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:41:39.729 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:41:39.729 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:41:39.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:41:39.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:41:39.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:41:49.732 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:41:49.766 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:41:49.766 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:41:49.766 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:41:49.766 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:41:49.766 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:41:49.766 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:41:49.766 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:41:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:41:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:41:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:42:09.770 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:42:09.803 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:42:09.803 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:42:09.803 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:42:09.803 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:42:09.803 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:42:09.803 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:42:09.803 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:42:09.805 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:42:09.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:42:09.806 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:42:09.806 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:42:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:42:09.806 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:42:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:42:09.806 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:42:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:42:09.806 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:42:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:42:09.806 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:42:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:42:09.806 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:42:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:42:09.806 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:42:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:42:09.806 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:42:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:42:09.806 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:42:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:42:09.806 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:42:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:42:09.806 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:42:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:42:09.806 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:42:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:42:09.806 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:42:09.806 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:42:09.806 139328 ERROR oslo_service.periodic_task >2018-06-24 21:42:39.688 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:42:39.689 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:42:39.696 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:42:39.697 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:42:39.731 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:42:39.731 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:42:39.731 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:42:39.731 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:42:39.731 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:42:39.731 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:42:39.731 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:42:39.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:42:39.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:42:39.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:42:49.734 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:42:49.768 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:42:49.768 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:42:49.768 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:42:49.768 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:42:49.768 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:42:49.768 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:42:49.768 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:42:49.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:42:49.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:42:49.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:43:09.771 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:43:09.807 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:43:09.807 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:43:09.807 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:43:09.807 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:43:09.807 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:43:09.807 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:43:09.807 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:43:09.809 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:43:09.810 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:43:09.810 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:43:09.810 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:43:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:43:09.810 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:43:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:43:09.810 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:43:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:43:09.810 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:43:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:43:09.810 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:43:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:43:09.810 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:43:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:43:09.810 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:43:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:43:09.810 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:43:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:43:09.810 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:43:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:43:09.810 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:43:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:43:09.810 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:43:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:43:09.810 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:43:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:43:09.810 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:43:09.810 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:43:09.810 139328 ERROR oslo_service.periodic_task >2018-06-24 21:43:39.691 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:43:39.693 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:43:39.700 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:43:39.700 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:43:39.741 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:43:39.741 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:43:39.741 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:43:39.741 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:43:39.741 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:43:39.741 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:43:39.741 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:43:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:43:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.043 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:43:39.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:43:49.744 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:43:49.777 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:43:49.777 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:43:49.777 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:43:49.777 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:43:49.777 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:43:49.777 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:43:49.777 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:43:49.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:43:49.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:43:49.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:44:09.780 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:44:09.814 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:44:09.814 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:44:09.814 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:44:09.814 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:44:09.814 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:44:09.814 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:44:09.814 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:44:09.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:44:09.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:44:09.817 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:44:09.817 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:44:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:44:09.817 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:44:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:44:09.817 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:44:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:44:09.817 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:44:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:44:09.817 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:44:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:44:09.817 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:44:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:44:09.817 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:44:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:44:09.817 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:44:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:44:09.817 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:44:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:44:09.817 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:44:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:44:09.817 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:44:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:44:09.817 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:44:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:44:09.817 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:44:09.817 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:44:09.817 139328 ERROR oslo_service.periodic_task >2018-06-24 21:44:39.688 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:44:39.689 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:44:39.696 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:44:39.696 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:44:39.733 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:44:39.733 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:44:39.733 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:44:39.733 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:44:39.733 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:44:39.733 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:44:39.733 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:44:39.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:44:39.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:44:39.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:44:49.738 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:44:49.773 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:44:49.773 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:44:49.773 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:44:49.773 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:44:49.773 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:44:49.773 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:44:49.773 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:44:49.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:44:49.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:44:49.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:45:09.776 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:45:09.812 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:45:09.812 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:45:09.812 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:45:09.812 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:45:09.812 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:45:09.812 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:45:09.812 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:45:09.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:45:09.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.119 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:45:09.817 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:45:09.817 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:45:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:45:09.817 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:45:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:45:09.817 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:45:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:45:09.817 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:45:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:45:09.817 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:45:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:45:09.817 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:45:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:45:09.817 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:45:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:45:09.817 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:45:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:45:09.817 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:45:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:45:09.817 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:45:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:45:09.817 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:45:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:45:09.817 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:45:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:45:09.817 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:45:09.817 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:45:09.817 139328 ERROR oslo_service.periodic_task >2018-06-24 21:45:39.689 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:45:39.690 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:45:39.697 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:45:39.698 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:45:39.746 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:45:39.746 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:45:39.746 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:45:39.746 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:45:39.746 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:45:39.746 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:45:39.746 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:45:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:45:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.052 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:45:39.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:45:49.752 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:45:49.787 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:45:49.787 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:45:49.787 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:45:49.787 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:45:49.787 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:45:49.787 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:45:49.787 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:45:49.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:45:49.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.092 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:45:49.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:46:09.791 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:46:09.845 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:46:09.845 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:46:09.845 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:46:09.845 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:46:09.845 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:46:09.845 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:46:09.845 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:46:09.847 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:46:09.848 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.15 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:46:09.848 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:46:09.848 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:46:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:46:09.848 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:46:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:46:09.848 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:46:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:46:09.848 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:46:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:46:09.848 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:46:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:46:09.848 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:46:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:46:09.848 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:46:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:46:09.848 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:46:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:46:09.848 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:46:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:46:09.848 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:46:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:46:09.848 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:46:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:46:09.848 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:46:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:46:09.848 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:46:09.848 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:46:09.848 139328 ERROR oslo_service.periodic_task >2018-06-24 21:46:39.690 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:46:39.691 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:46:39.697 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:46:39.698 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:46:39.736 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:46:39.736 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:46:39.736 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:46:39.736 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:46:39.736 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:46:39.736 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:46:39.736 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:46:39.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:46:39.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.041 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:46:39.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:46:49.740 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:46:49.772 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:46:49.772 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:46:49.772 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:46:49.772 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:46:49.772 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:46:49.772 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:46:49.772 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:46:49.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:46:49.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:46:49.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:47:09.776 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:47:09.809 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:47:09.809 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:47:09.809 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:47:09.809 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:47:09.809 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:47:09.809 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:47:09.809 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:47:09.811 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:47:09.811 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:47:09.812 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:47:09.812 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:47:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:47:09.812 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:47:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:47:09.812 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:47:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:47:09.812 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:47:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:47:09.812 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:47:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:47:09.812 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:47:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:47:09.812 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:47:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:47:09.812 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:47:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:47:09.812 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:47:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:47:09.812 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:47:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:47:09.812 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:47:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:47:09.812 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:47:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:47:09.812 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:47:09.812 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:47:09.812 139328 ERROR oslo_service.periodic_task >2018-06-24 21:47:39.693 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:47:39.694 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:47:39.706 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:47:39.707 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:47:39.759 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:47:39.759 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:47:39.759 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:47:39.759 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:47:39.759 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:47:39.759 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:47:39.759 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:47:39.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:47:39.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.054 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:47:39.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:47:49.762 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:47:49.795 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:47:49.795 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:47:49.795 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:47:49.795 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:47:49.795 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:47:49.795 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:47:49.795 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:47:49.797 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:47:49.798 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.091 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:47:49.798 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:48:09.798 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:48:09.832 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:48:09.832 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:48:09.832 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:48:09.832 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:48:09.832 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:48:09.832 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:48:09.832 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:48:09.834 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:48:09.834 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.128 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:48:09.835 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:48:09.835 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:48:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:48:09.835 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:48:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:48:09.835 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:48:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:48:09.835 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:48:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:48:09.835 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:48:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:48:09.835 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:48:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:48:09.835 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:48:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:48:09.835 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:48:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:48:09.835 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:48:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:48:09.835 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:48:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:48:09.835 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:48:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:48:09.835 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:48:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:48:09.835 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:48:09.835 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:48:09.835 139328 ERROR oslo_service.periodic_task >2018-06-24 21:48:39.696 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:48:39.697 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:48:39.704 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:48:39.704 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:48:39.737 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:48:39.737 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:48:39.737 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:48:39.737 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:48:39.737 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:48:39.737 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:48:39.737 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:48:39.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:48:39.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:48:39.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:48:49.740 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:48:49.774 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:48:49.774 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:48:49.774 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:48:49.774 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:48:49.774 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:48:49.774 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:48:49.774 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:48:49.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:48:49.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:48:49.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:49:09.777 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:49:09.813 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:49:09.813 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:49:09.813 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:49:09.813 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:49:09.813 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:49:09.813 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:49:09.813 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:49:09.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:49:09.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:49:09.816 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:49:09.816 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:49:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:49:09.816 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:49:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:49:09.816 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:49:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:49:09.816 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:49:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:49:09.816 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:49:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:49:09.816 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:49:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:49:09.816 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:49:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:49:09.816 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:49:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:49:09.816 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:49:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:49:09.816 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:49:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:49:09.816 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:49:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:49:09.816 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:49:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:49:09.816 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:49:09.816 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:49:09.816 139328 ERROR oslo_service.periodic_task >2018-06-24 21:49:39.698 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:49:39.699 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:49:39.706 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:49:39.707 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:49:39.740 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:49:39.740 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:49:39.740 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:49:39.740 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:49:39.740 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:49:39.740 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:49:39.740 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:49:39.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:49:39.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:49:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:49:49.743 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:49:49.776 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:49:49.776 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:49:49.776 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:49:49.776 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:49:49.776 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:49:49.776 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:49:49.776 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:49:49.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:49:49.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:49:49.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:50:09.779 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:50:09.814 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:50:09.814 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:50:09.814 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:50:09.814 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:50:09.814 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:50:09.814 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:50:09.814 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:50:09.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:50:09.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:50:09.817 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:50:09.817 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:50:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:50:09.817 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:50:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:50:09.817 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:50:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:50:09.817 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:50:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:50:09.817 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:50:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:50:09.817 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:50:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:50:09.817 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:50:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:50:09.817 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:50:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:50:09.817 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:50:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:50:09.817 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:50:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:50:09.817 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:50:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:50:09.817 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:50:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:50:09.817 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:50:09.817 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:50:09.817 139328 ERROR oslo_service.periodic_task >2018-06-24 21:50:39.698 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:50:39.699 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:50:39.707 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:50:39.707 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:50:39.742 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:50:39.742 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:50:39.742 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:50:39.742 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:50:39.742 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:50:39.742 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:50:39.742 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:50:39.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:50:39.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:50:39.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:50:49.746 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:50:49.798 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:50:49.798 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:50:49.798 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:50:49.798 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:50:49.798 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:50:49.798 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:50:49.798 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:50:49.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:50:49.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.092 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:50:49.801 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:51:09.801 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:51:09.836 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:51:09.836 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:51:09.836 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:51:09.836 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:51:09.836 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:51:09.836 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:51:09.836 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:51:09.838 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:51:09.838 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.131 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:51:09.839 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:51:09.839 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:51:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:51:09.839 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:51:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:51:09.839 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:51:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:51:09.839 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:51:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:51:09.839 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:51:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:51:09.839 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:51:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:51:09.839 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:51:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:51:09.839 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:51:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:51:09.839 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:51:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:51:09.839 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:51:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:51:09.839 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:51:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:51:09.839 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:51:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:51:09.839 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:51:09.839 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:51:09.839 139328 ERROR oslo_service.periodic_task >2018-06-24 21:51:39.700 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:51:39.701 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:51:39.708 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:51:39.709 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:51:39.743 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:51:39.743 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:51:39.743 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:51:39.743 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:51:39.743 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:51:39.743 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:51:39.743 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:51:39.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:51:39.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:51:39.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:51:49.746 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:51:49.780 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:51:49.780 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:51:49.780 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:51:49.780 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:51:49.780 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:51:49.780 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:51:49.780 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:51:49.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:51:49.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:51:49.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:52:09.783 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:52:09.819 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:52:09.819 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:52:09.819 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:52:09.819 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:52:09.819 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:52:09.819 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:52:09.819 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:52:09.821 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:52:09.821 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:52:09.822 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:52:09.822 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:52:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:52:09.822 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:52:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:52:09.822 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:52:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:52:09.822 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:52:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:52:09.822 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:52:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:52:09.822 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:52:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:52:09.822 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:52:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:52:09.822 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:52:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:52:09.822 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:52:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:52:09.822 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:52:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:52:09.822 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:52:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:52:09.822 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:52:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:52:09.822 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:52:09.822 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:52:09.822 139328 ERROR oslo_service.periodic_task >2018-06-24 21:52:39.703 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:52:39.704 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:52:39.712 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:52:39.713 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:52:39.752 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:52:39.752 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:52:39.752 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:52:39.752 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:52:39.752 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:52:39.752 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:52:39.752 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:52:39.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:52:39.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.042 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:52:39.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:52:49.756 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:52:49.791 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:52:49.791 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:52:49.791 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:52:49.791 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:52:49.791 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:52:49.791 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:52:49.791 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:52:49.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:52:49.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.08 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:52:49.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:53:09.794 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:53:09.827 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:53:09.827 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:53:09.827 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:53:09.827 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:53:09.827 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:53:09.827 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:53:09.827 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:53:09.829 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:53:09.830 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:53:09.830 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:53:09.830 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:53:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:53:09.830 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:53:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:53:09.830 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:53:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:53:09.830 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:53:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:53:09.830 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:53:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:53:09.830 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:53:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:53:09.830 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:53:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:53:09.830 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:53:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:53:09.830 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:53:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:53:09.830 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:53:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:53:09.830 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:53:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:53:09.830 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:53:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:53:09.830 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:53:09.830 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:53:09.830 139328 ERROR oslo_service.periodic_task >2018-06-24 21:53:39.701 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:53:39.703 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:53:39.710 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:53:39.710 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:53:39.748 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:53:39.748 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:53:39.748 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:53:39.748 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:53:39.748 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:53:39.748 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:53:39.748 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:53:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:53:39.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:53:39.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:53:49.752 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:53:49.786 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:53:49.786 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:53:49.786 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:53:49.786 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:53:49.786 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:53:49.786 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:53:49.786 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:53:49.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:53:49.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:53:49.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:54:09.790 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:54:09.824 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:54:09.824 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:54:09.824 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:54:09.824 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:54:09.824 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:54:09.824 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:54:09.824 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:54:09.826 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:54:09.826 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:54:09.827 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:54:09.827 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:54:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:54:09.827 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:54:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:54:09.827 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:54:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:54:09.827 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:54:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:54:09.827 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:54:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:54:09.827 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:54:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:54:09.827 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:54:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:54:09.827 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:54:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:54:09.827 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:54:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:54:09.827 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:54:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:54:09.827 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:54:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:54:09.827 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:54:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:54:09.827 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:54:09.827 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:54:09.827 139328 ERROR oslo_service.periodic_task >2018-06-24 21:54:39.698 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:54:39.699 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:54:39.740 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:54:39.740 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:54:39.740 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:54:39.740 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:54:39.740 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:54:39.740 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:54:39.740 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:54:39.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:54:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.043 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:54:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:54:49.743 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:54:49.776 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:54:49.776 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:54:49.776 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:54:49.776 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:54:49.776 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:54:49.776 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:54:49.776 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:54:49.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:54:49.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:54:49.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:55:09.781 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:55:09.833 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:55:09.833 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:55:09.833 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:55:09.833 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:55:09.833 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:55:09.833 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:55:09.833 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:55:09.835 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:55:09.835 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.135 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:55:09.836 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:55:09.836 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:55:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:55:09.836 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:55:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:55:09.836 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:55:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:55:09.836 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:55:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:55:09.836 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:55:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:55:09.836 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:55:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:55:09.836 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:55:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:55:09.836 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:55:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:55:09.836 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:55:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:55:09.836 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:55:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:55:09.836 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:55:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:55:09.836 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:55:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:55:09.836 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:55:09.836 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:55:09.836 139328 ERROR oslo_service.periodic_task >2018-06-24 21:55:39.697 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:55:39.698 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:55:39.705 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:55:39.706 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:55:39.741 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:55:39.741 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:55:39.741 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:55:39.741 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:55:39.741 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:55:39.741 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:55:39.741 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:55:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:55:39.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:55:39.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:55:49.745 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:55:49.781 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:55:49.781 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:55:49.781 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:55:49.781 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:55:49.781 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:55:49.781 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:55:49.781 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:55:49.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:55:49.783 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:55:49.783 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:56:09.784 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:56:09.818 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:56:09.818 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:56:09.818 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:56:09.818 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:56:09.818 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:56:09.818 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:56:09.818 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:56:09.820 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:56:09.820 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:56:09.821 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:56:09.821 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:56:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:56:09.821 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:56:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:56:09.821 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:56:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:56:09.821 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:56:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:56:09.821 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:56:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:56:09.821 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:56:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:56:09.821 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:56:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:56:09.821 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:56:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:56:09.821 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:56:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:56:09.821 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:56:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:56:09.821 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:56:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:56:09.821 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:56:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:56:09.821 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:56:09.821 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:56:09.821 139328 ERROR oslo_service.periodic_task >2018-06-24 21:56:39.702 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:56:39.703 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:56:39.716 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:56:39.717 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:56:39.750 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:56:39.750 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:56:39.750 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:56:39.750 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:56:39.750 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:56:39.750 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:56:39.750 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:56:39.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:56:39.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:56:39.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:56:49.753 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:56:49.794 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:56:49.794 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:56:49.794 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:56:49.794 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:56:49.794 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:56:49.794 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:56:49.794 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:56:49.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:56:49.797 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.08 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:56:49.797 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:57:09.797 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:57:09.837 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:57:09.837 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:57:09.837 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:57:09.837 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:57:09.837 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:57:09.837 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:57:09.837 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:57:09.839 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:57:09.839 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.122 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:57:09.840 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:57:09.840 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:57:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:57:09.840 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:57:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:57:09.840 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:57:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:57:09.840 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:57:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:57:09.840 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:57:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:57:09.840 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:57:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:57:09.840 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:57:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:57:09.840 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:57:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:57:09.840 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:57:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:57:09.840 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:57:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:57:09.840 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:57:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:57:09.840 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:57:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:57:09.840 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:57:09.840 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:57:09.840 139328 ERROR oslo_service.periodic_task >2018-06-24 21:57:39.702 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:57:39.703 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:57:39.710 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:57:39.710 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:57:39.746 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:57:39.746 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:57:39.746 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:57:39.746 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:57:39.746 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:57:39.746 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:57:39.746 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:57:39.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:57:39.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:57:39.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:57:49.750 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:57:49.784 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:57:49.784 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:57:49.784 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:57:49.784 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:57:49.784 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:57:49.784 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:57:49.784 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:57:49.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:57:49.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:57:49.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:58:09.787 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:58:09.821 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:58:09.821 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:58:09.821 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:58:09.821 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:58:09.821 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:58:09.821 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:58:09.821 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:58:09.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:58:09.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:58:09.823 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:58:09.823 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:58:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:58:09.823 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:58:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:58:09.823 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:58:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:58:09.823 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:58:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:58:09.823 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:58:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:58:09.823 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:58:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:58:09.823 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:58:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:58:09.823 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:58:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:58:09.823 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:58:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:58:09.823 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:58:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:58:09.823 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:58:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:58:09.823 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:58:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:58:09.823 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:58:09.823 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:58:09.823 139328 ERROR oslo_service.periodic_task >2018-06-24 21:58:39.705 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:58:39.706 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:58:39.713 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:58:39.713 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:58:39.746 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:58:39.746 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:58:39.746 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:58:39.746 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:58:39.746 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:58:39.746 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:58:39.746 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:58:39.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:58:39.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:58:39.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:58:49.750 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:58:49.784 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:58:49.784 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:58:49.784 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:58:49.784 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:58:49.784 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:58:49.784 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:58:49.784 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:58:49.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:58:49.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:58:49.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:59:09.787 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:59:09.820 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:59:09.820 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:59:09.820 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:59:09.820 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:59:09.820 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:59:09.820 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:59:09.820 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:59:09.821 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:59:09.822 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:59:09.822 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:59:09.822 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 21:59:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 21:59:09.822 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 21:59:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 21:59:09.822 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 21:59:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 21:59:09.822 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 21:59:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 21:59:09.822 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 21:59:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 21:59:09.822 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 21:59:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 21:59:09.822 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 21:59:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 21:59:09.822 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 21:59:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 21:59:09.822 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 21:59:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 21:59:09.822 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 21:59:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 21:59:09.822 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 21:59:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 21:59:09.822 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 21:59:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 21:59:09.822 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 21:59:09.822 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 21:59:09.822 139328 ERROR oslo_service.periodic_task >2018-06-24 21:59:39.704 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:59:39.705 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 21:59:39.711 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 21:59:39.711 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:59:39.745 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:59:39.745 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:59:39.745 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:59:39.745 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:59:39.745 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:59:39.745 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:59:39.745 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:59:39.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:59:39.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:59:39.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 21:59:49.748 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 21:59:49.788 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 21:59:49.788 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 21:59:49.788 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 21:59:49.788 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 21:59:49.788 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 21:59:49.788 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 21:59:49.788 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 21:59:49.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 21:59:49.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 21:59:49.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:00:09.791 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:00:09.826 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:00:09.826 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:00:09.826 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:00:09.826 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:00:09.826 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:00:09.826 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:00:09.826 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:00:09.828 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:00:09.829 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:00:09.829 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:00:09.829 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:00:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:00:09.829 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:00:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:00:09.829 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:00:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:00:09.829 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:00:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:00:09.829 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:00:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:00:09.829 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:00:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:00:09.829 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:00:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:00:09.829 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:00:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:00:09.829 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:00:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:00:09.829 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:00:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:00:09.829 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:00:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:00:09.829 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:00:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:00:09.829 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:00:09.829 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:00:09.829 139328 ERROR oslo_service.periodic_task >2018-06-24 22:00:39.700 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:00:39.701 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:00:39.708 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:00:39.709 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:00:39.743 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:00:39.743 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:00:39.743 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:00:39.743 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:00:39.743 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:00:39.743 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:00:39.743 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:00:39.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:00:39.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:00:39.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:00:49.746 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:00:49.780 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:00:49.780 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:00:49.780 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:00:49.780 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:00:49.780 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:00:49.780 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:00:49.780 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:00:49.783 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:00:49.783 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:00:49.783 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:01:09.784 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:01:09.819 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:01:09.819 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:01:09.819 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:01:09.819 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:01:09.819 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:01:09.819 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:01:09.819 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:01:09.821 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:01:09.821 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:01:09.821 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:01:09.821 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:01:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:01:09.821 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:01:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:01:09.821 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:01:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:01:09.821 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:01:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:01:09.821 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:01:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:01:09.821 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:01:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:01:09.821 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:01:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:01:09.821 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:01:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:01:09.821 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:01:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:01:09.821 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:01:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:01:09.821 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:01:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:01:09.821 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:01:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:01:09.821 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:01:09.821 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:01:09.821 139328 ERROR oslo_service.periodic_task >2018-06-24 22:01:39.703 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:01:39.704 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:01:39.711 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:01:39.712 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:01:39.748 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:01:39.748 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:01:39.748 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:01:39.748 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:01:39.748 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:01:39.748 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:01:39.748 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:01:39.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:01:39.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:01:39.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:01:49.754 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:01:49.787 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:01:49.787 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:01:49.787 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:01:49.787 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:01:49.787 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:01:49.787 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:01:49.787 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:01:49.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:01:49.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:01:49.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:02:09.790 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:02:09.825 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:02:09.825 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:02:09.825 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:02:09.825 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:02:09.825 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:02:09.825 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:02:09.825 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:02:09.827 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:02:09.827 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:02:09.827 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:02:09.827 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:02:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:02:09.827 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:02:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:02:09.827 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:02:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:02:09.827 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:02:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:02:09.827 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:02:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:02:09.827 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:02:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:02:09.827 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:02:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:02:09.827 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:02:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:02:09.827 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:02:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:02:09.827 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:02:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:02:09.827 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:02:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:02:09.827 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:02:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:02:09.827 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:02:09.827 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:02:09.827 139328 ERROR oslo_service.periodic_task >2018-06-24 22:02:39.699 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:02:39.700 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:02:39.707 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:02:39.707 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:02:39.746 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:02:39.746 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:02:39.746 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:02:39.746 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:02:39.746 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:02:39.746 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:02:39.746 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:02:39.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:02:39.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.041 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:02:39.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:02:49.750 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:02:49.797 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:02:49.797 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:02:49.797 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:02:49.797 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:02:49.797 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:02:49.797 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:02:49.797 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:02:49.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:02:49.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.092 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:02:49.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:03:09.801 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:03:09.835 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:03:09.835 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:03:09.835 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:03:09.835 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:03:09.835 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:03:09.835 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:03:09.835 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:03:09.837 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:03:09.837 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.13 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:03:09.838 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:03:09.838 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:03:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:03:09.838 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:03:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:03:09.838 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:03:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:03:09.838 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:03:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:03:09.838 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:03:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:03:09.838 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:03:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:03:09.838 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:03:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:03:09.838 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:03:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:03:09.838 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:03:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:03:09.838 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:03:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:03:09.838 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:03:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:03:09.838 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:03:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:03:09.838 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:03:09.838 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:03:09.838 139328 ERROR oslo_service.periodic_task >2018-06-24 22:03:39.699 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:03:39.700 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:03:39.707 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:03:39.707 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:03:39.741 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:03:39.741 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:03:39.741 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:03:39.741 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:03:39.741 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:03:39.741 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:03:39.741 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:03:39.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:03:39.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:03:39.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:03:49.745 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:03:49.779 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:03:49.779 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:03:49.779 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:03:49.779 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:03:49.779 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:03:49.779 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:03:49.779 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:03:49.781 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:03:49.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:03:49.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:04:09.783 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:04:09.817 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:04:09.817 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:04:09.817 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:04:09.817 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:04:09.817 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:04:09.817 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:04:09.817 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:04:09.819 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:04:09.819 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:04:09.820 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:04:09.820 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:04:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:04:09.820 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:04:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:04:09.820 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:04:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:04:09.820 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:04:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:04:09.820 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:04:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:04:09.820 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:04:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:04:09.820 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:04:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:04:09.820 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:04:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:04:09.820 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:04:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:04:09.820 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:04:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:04:09.820 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:04:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:04:09.820 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:04:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:04:09.820 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:04:09.820 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:04:09.820 139328 ERROR oslo_service.periodic_task >2018-06-24 22:04:39.701 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:04:39.702 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:04:39.710 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:04:39.711 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:04:39.745 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:04:39.745 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:04:39.745 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:04:39.745 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:04:39.745 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:04:39.745 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:04:39.745 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:04:39.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:04:39.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:04:39.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:04:49.749 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:04:49.782 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:04:49.782 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:04:49.782 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:04:49.782 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:04:49.782 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:04:49.782 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:04:49.782 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:04:49.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:04:49.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:04:49.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:05:09.785 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:05:09.819 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:05:09.819 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:05:09.819 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:05:09.819 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:05:09.819 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:05:09.819 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:05:09.819 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:05:09.821 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:05:09.821 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:05:09.821 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:05:09.821 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:05:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:05:09.821 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:05:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:05:09.821 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:05:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:05:09.821 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:05:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:05:09.821 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:05:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:05:09.821 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:05:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:05:09.821 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:05:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:05:09.821 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:05:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:05:09.821 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:05:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:05:09.821 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:05:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:05:09.821 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:05:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:05:09.821 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:05:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:05:09.821 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:05:09.821 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:05:09.821 139328 ERROR oslo_service.periodic_task >2018-06-24 22:05:39.702 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:05:39.704 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:05:39.712 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:05:39.713 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:05:39.746 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:05:39.746 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:05:39.746 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:05:39.746 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:05:39.746 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:05:39.746 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:05:39.746 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:05:39.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:05:39.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:05:39.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:05:49.749 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:05:49.783 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:05:49.783 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:05:49.783 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:05:49.783 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:05:49.783 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:05:49.783 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:05:49.783 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:05:49.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:05:49.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:05:49.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:06:09.787 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:06:09.819 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:06:09.819 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:06:09.819 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:06:09.819 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:06:09.819 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:06:09.819 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:06:09.819 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:06:09.821 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:06:09.822 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:06:09.822 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:06:09.822 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:06:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:06:09.822 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:06:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:06:09.822 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:06:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:06:09.822 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:06:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:06:09.822 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:06:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:06:09.822 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:06:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:06:09.822 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:06:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:06:09.822 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:06:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:06:09.822 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:06:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:06:09.822 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:06:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:06:09.822 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:06:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:06:09.822 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:06:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:06:09.822 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:06:09.822 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:06:09.822 139328 ERROR oslo_service.periodic_task >2018-06-24 22:06:39.703 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:06:39.704 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:06:39.711 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:06:39.712 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:06:39.744 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:06:39.744 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:06:39.744 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:06:39.744 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:06:39.744 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:06:39.744 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:06:39.744 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:06:39.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:06:39.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:06:39.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:06:49.748 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:06:49.782 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:06:49.782 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:06:49.782 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:06:49.782 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:06:49.782 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:06:49.782 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:06:49.782 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:06:49.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:06:49.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:06:49.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:07:09.787 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:07:09.820 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:07:09.820 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:07:09.820 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:07:09.820 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:07:09.820 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:07:09.820 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:07:09.820 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:07:09.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:07:09.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:07:09.823 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:07:09.823 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:07:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:07:09.823 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:07:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:07:09.823 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:07:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:07:09.823 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:07:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:07:09.823 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:07:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:07:09.823 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:07:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:07:09.823 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:07:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:07:09.823 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:07:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:07:09.823 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:07:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:07:09.823 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:07:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:07:09.823 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:07:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:07:09.823 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:07:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:07:09.823 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:07:09.823 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:07:09.823 139328 ERROR oslo_service.periodic_task >2018-06-24 22:07:39.705 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:07:39.706 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:07:39.713 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:07:39.713 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:07:39.745 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:07:39.745 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:07:39.745 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:07:39.745 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:07:39.745 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:07:39.745 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:07:39.745 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:07:39.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:07:39.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:07:39.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:07:49.748 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:07:49.786 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:07:49.786 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:07:49.786 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:07:49.786 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:07:49.786 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:07:49.786 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:07:49.786 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:07:49.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:07:49.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:07:49.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:08:09.790 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:08:09.825 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:08:09.825 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:08:09.825 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:08:09.825 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:08:09.825 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:08:09.825 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:08:09.825 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:08:09.828 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:08:09.828 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:08:09.828 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:08:09.828 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:08:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:08:09.828 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:08:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:08:09.828 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:08:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:08:09.828 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:08:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:08:09.828 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:08:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:08:09.828 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:08:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:08:09.828 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:08:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:08:09.828 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:08:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:08:09.828 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:08:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:08:09.828 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:08:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:08:09.828 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:08:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:08:09.828 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:08:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:08:09.828 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:08:09.828 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:08:09.828 139328 ERROR oslo_service.periodic_task >2018-06-24 22:08:39.700 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:08:39.701 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:08:39.708 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:08:39.709 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:08:39.745 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:08:39.745 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:08:39.745 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:08:39.745 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:08:39.745 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:08:39.745 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:08:39.745 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:08:39.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:08:39.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:08:39.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:08:49.748 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:08:49.782 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:08:49.782 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:08:49.782 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:08:49.782 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:08:49.782 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:08:49.782 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:08:49.782 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:08:49.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:08:49.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:08:49.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:09:09.785 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:09:09.820 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:09:09.820 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:09:09.820 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:09:09.820 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:09:09.820 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:09:09.820 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:09:09.820 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:09:09.822 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:09:09.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:09:09.823 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:09:09.823 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:09:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:09:09.823 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:09:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:09:09.823 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:09:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:09:09.823 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:09:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:09:09.823 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:09:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:09:09.823 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:09:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:09:09.823 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:09:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:09:09.823 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:09:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:09:09.823 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:09:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:09:09.823 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:09:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:09:09.823 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:09:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:09:09.823 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:09:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:09:09.823 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:09:09.823 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:09:09.823 139328 ERROR oslo_service.periodic_task >2018-06-24 22:09:39.705 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:09:39.706 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:09:39.720 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:09:39.721 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:09:39.754 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:09:39.754 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:09:39.754 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:09:39.754 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:09:39.754 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:09:39.754 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:09:39.754 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:09:39.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:09:39.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:09:39.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:09:49.757 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:09:49.788 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:09:49.788 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:09:49.788 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:09:49.788 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:09:49.788 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:09:49.788 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:09:49.788 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:09:49.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:09:49.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.07 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:09:49.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:10:09.791 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:10:09.826 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:10:09.826 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:10:09.826 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:10:09.826 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:10:09.826 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:10:09.826 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:10:09.826 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:10:09.829 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:10:09.829 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:10:09.829 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:10:09.829 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:10:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:10:09.829 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:10:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:10:09.829 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:10:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:10:09.829 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:10:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:10:09.829 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:10:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:10:09.829 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:10:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:10:09.829 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:10:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:10:09.829 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:10:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:10:09.829 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:10:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:10:09.829 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:10:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:10:09.829 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:10:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:10:09.829 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:10:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:10:09.829 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:10:09.829 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:10:09.829 139328 ERROR oslo_service.periodic_task >2018-06-24 22:10:39.701 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:10:39.702 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:10:39.708 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:10:39.708 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:10:39.741 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:10:39.741 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:10:39.741 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:10:39.741 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:10:39.741 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:10:39.741 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:10:39.741 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:10:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:10:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:10:39.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:10:49.744 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:10:49.777 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:10:49.777 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:10:49.777 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:10:49.777 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:10:49.777 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:10:49.777 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:10:49.777 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:10:49.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:10:49.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:10:49.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:11:09.780 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:11:09.814 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:11:09.814 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:11:09.814 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:11:09.814 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:11:09.814 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:11:09.814 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:11:09.814 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:11:09.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:11:09.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:11:09.817 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:11:09.817 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:11:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:11:09.817 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:11:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:11:09.817 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:11:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:11:09.817 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:11:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:11:09.817 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:11:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:11:09.817 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:11:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:11:09.817 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:11:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:11:09.817 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:11:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:11:09.817 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:11:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:11:09.817 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:11:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:11:09.817 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:11:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:11:09.817 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:11:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:11:09.817 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:11:09.817 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:11:09.817 139328 ERROR oslo_service.periodic_task >2018-06-24 22:11:39.699 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:11:39.700 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:11:39.708 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:11:39.709 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:11:39.759 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:11:39.759 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:11:39.759 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:11:39.759 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:11:39.759 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:11:39.759 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:11:39.759 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:11:39.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:11:39.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.053 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:11:39.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:11:49.763 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:11:49.797 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:11:49.797 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:11:49.797 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:11:49.797 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:11:49.797 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:11:49.797 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:11:49.797 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:11:49.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:11:49.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.091 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:11:49.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:12:09.801 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:12:09.837 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:12:09.837 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:12:09.837 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:12:09.837 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:12:09.837 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:12:09.837 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:12:09.837 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:12:09.839 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:12:09.839 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.131 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:12:09.840 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:12:09.840 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:12:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:12:09.840 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:12:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:12:09.840 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:12:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:12:09.840 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:12:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:12:09.840 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:12:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:12:09.840 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:12:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:12:09.840 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:12:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:12:09.840 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:12:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:12:09.840 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:12:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:12:09.840 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:12:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:12:09.840 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:12:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:12:09.840 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:12:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:12:09.840 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:12:09.840 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:12:09.840 139328 ERROR oslo_service.periodic_task >2018-06-24 22:12:39.701 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:12:39.702 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:12:39.709 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:12:39.709 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:12:39.743 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:12:39.743 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:12:39.743 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:12:39.743 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:12:39.743 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:12:39.743 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:12:39.743 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:12:39.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:12:39.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:12:39.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:12:49.746 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:12:49.783 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:12:49.783 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:12:49.783 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:12:49.783 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:12:49.783 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:12:49.783 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:12:49.783 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:12:49.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:12:49.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:12:49.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:13:09.787 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:13:09.821 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:13:09.821 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:13:09.821 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:13:09.821 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:13:09.821 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:13:09.821 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:13:09.821 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:13:09.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:13:09.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:13:09.824 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:13:09.824 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:13:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:13:09.824 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:13:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:13:09.824 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:13:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:13:09.824 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:13:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:13:09.824 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:13:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:13:09.824 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:13:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:13:09.824 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:13:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:13:09.824 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:13:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:13:09.824 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:13:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:13:09.824 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:13:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:13:09.824 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:13:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:13:09.824 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:13:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:13:09.824 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:13:09.824 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:13:09.824 139328 ERROR oslo_service.periodic_task >2018-06-24 22:13:39.705 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:13:39.706 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:13:39.715 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:13:39.715 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:13:39.759 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:13:39.759 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:13:39.759 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:13:39.759 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:13:39.759 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:13:39.759 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:13:39.759 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:13:39.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:13:39.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.045 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:13:39.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:13:49.762 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:13:49.799 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:13:49.799 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:13:49.799 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:13:49.799 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:13:49.799 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:13:49.799 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:13:49.799 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:13:49.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:13:49.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.086 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:13:49.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:14:09.803 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:14:09.836 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:14:09.836 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:14:09.836 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:14:09.836 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:14:09.836 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:14:09.836 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:14:09.836 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:14:09.838 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:14:09.838 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.122 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:14:09.838 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:14:09.838 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:14:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:14:09.838 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:14:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:14:09.838 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:14:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:14:09.838 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:14:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:14:09.838 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:14:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:14:09.838 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:14:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:14:09.838 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:14:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:14:09.838 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:14:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:14:09.838 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:14:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:14:09.838 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:14:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:14:09.838 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:14:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:14:09.838 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:14:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:14:09.838 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:14:09.838 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:14:09.838 139328 ERROR oslo_service.periodic_task >2018-06-24 22:14:39.710 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:14:39.711 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:14:39.719 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:14:39.720 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:14:39.782 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:14:39.782 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:14:39.782 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:14:39.782 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:14:39.782 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:14:39.782 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:14:39.782 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:14:39.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:14:39.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.066 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:14:39.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:14:49.787 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:14:49.822 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:14:49.822 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:14:49.822 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:14:49.822 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:14:49.822 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:14:49.822 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:14:49.822 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:14:49.825 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:14:49.825 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.105 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:14:49.825 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:15:09.826 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:15:09.862 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:15:09.862 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:15:09.862 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:15:09.862 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:15:09.862 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:15:09.862 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:15:09.862 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:15:09.865 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:15:09.865 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.145 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:15:09.865 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:15:09.865 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:15:09.865 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:15:09.865 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:15:09.865 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:15:09.865 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:15:09.865 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:15:09.865 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:15:09.865 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:15:09.865 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:15:09.865 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:15:09.865 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:15:09.865 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:15:09.865 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:15:09.865 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:15:09.865 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:15:09.865 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:15:09.865 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:15:09.865 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:15:09.865 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:15:09.865 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:15:09.865 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:15:09.865 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:15:09.865 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:15:09.865 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:15:09.865 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:15:09.865 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:15:09.865 139328 ERROR oslo_service.periodic_task >2018-06-24 22:15:39.707 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:15:39.708 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:15:39.715 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:15:39.716 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:15:39.751 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:15:39.751 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:15:39.751 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:15:39.751 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:15:39.751 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:15:39.751 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:15:39.751 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:15:39.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:15:39.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:15:39.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:15:49.755 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:15:49.790 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:15:49.790 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:15:49.790 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:15:49.790 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:15:49.790 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:15:49.790 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:15:49.790 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:15:49.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:15:49.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:15:49.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:16:09.794 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:16:09.828 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:16:09.828 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:16:09.828 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:16:09.828 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:16:09.828 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:16:09.828 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:16:09.828 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:16:09.830 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:16:09.830 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:16:09.831 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:16:09.831 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:16:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:16:09.831 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:16:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:16:09.831 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:16:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:16:09.831 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:16:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:16:09.831 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:16:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:16:09.831 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:16:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:16:09.831 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:16:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:16:09.831 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:16:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:16:09.831 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:16:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:16:09.831 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:16:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:16:09.831 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:16:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:16:09.831 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:16:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:16:09.831 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:16:09.831 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:16:09.831 139328 ERROR oslo_service.periodic_task >2018-06-24 22:16:39.702 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:16:39.704 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:16:39.710 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:16:39.711 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:16:39.747 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:16:39.747 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:16:39.747 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:16:39.747 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:16:39.747 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:16:39.747 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:16:39.747 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:16:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:16:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:16:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:16:49.751 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:16:49.784 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:16:49.784 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:16:49.784 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:16:49.784 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:16:49.784 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:16:49.784 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:16:49.784 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:16:49.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:16:49.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:16:49.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:17:09.787 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:17:09.825 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:17:09.825 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:17:09.825 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:17:09.825 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:17:09.825 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:17:09.825 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:17:09.825 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:17:09.827 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:17:09.827 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:17:09.828 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:17:09.828 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:17:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:17:09.828 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:17:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:17:09.828 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:17:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:17:09.828 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:17:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:17:09.828 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:17:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:17:09.828 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:17:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:17:09.828 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:17:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:17:09.828 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:17:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:17:09.828 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:17:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:17:09.828 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:17:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:17:09.828 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:17:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:17:09.828 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:17:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:17:09.828 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:17:09.828 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:17:09.828 139328 ERROR oslo_service.periodic_task >2018-06-24 22:17:39.699 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:17:39.700 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:17:39.711 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:17:39.712 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:17:39.756 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:17:39.756 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:17:39.756 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:17:39.756 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:17:39.756 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:17:39.756 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:17:39.756 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:17:39.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:17:39.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.045 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:17:39.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:17:49.759 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:17:49.793 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:17:49.793 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:17:49.793 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:17:49.793 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:17:49.793 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:17:49.793 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:17:49.793 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:17:49.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:17:49.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.082 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:17:49.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:18:09.796 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:18:09.834 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:18:09.834 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:18:09.834 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:18:09.834 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:18:09.834 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:18:09.834 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:18:09.834 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:18:09.837 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:18:09.837 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.124 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:18:09.837 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:18:09.837 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:18:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:18:09.837 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:18:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:18:09.837 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:18:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:18:09.837 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:18:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:18:09.837 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:18:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:18:09.837 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:18:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:18:09.837 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:18:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:18:09.837 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:18:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:18:09.837 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:18:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:18:09.837 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:18:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:18:09.837 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:18:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:18:09.837 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:18:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:18:09.837 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:18:09.837 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:18:09.837 139328 ERROR oslo_service.periodic_task >2018-06-24 22:18:39.699 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:18:39.700 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:18:39.709 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:18:39.710 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:18:39.763 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:18:39.763 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:18:39.763 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:18:39.763 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:18:39.763 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:18:39.763 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:18:39.763 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:18:39.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:18:39.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.055 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:18:39.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:18:49.767 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:18:49.801 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:18:49.801 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:18:49.801 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:18:49.801 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:18:49.801 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:18:49.801 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:18:49.801 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:18:49.803 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:18:49.803 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.092 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:18:49.804 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:19:09.804 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:19:09.838 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:19:09.838 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:19:09.838 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:19:09.838 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:19:09.838 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:19:09.838 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:19:09.838 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:19:09.841 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:19:09.842 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.131 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:19:09.842 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:19:09.842 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:19:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:19:09.842 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:19:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:19:09.842 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:19:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:19:09.842 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:19:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:19:09.842 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:19:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:19:09.842 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:19:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:19:09.842 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:19:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:19:09.842 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:19:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:19:09.842 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:19:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:19:09.842 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:19:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:19:09.842 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:19:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:19:09.842 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:19:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:19:09.842 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:19:09.842 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:19:09.842 139328 ERROR oslo_service.periodic_task >2018-06-24 22:19:39.704 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:19:39.705 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:19:39.711 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:19:39.712 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:19:39.746 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:19:39.746 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:19:39.746 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:19:39.746 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:19:39.746 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:19:39.746 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:19:39.746 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:19:39.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:19:39.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:19:39.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:19:49.749 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:19:49.795 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:19:49.795 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:19:49.795 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:19:49.795 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:19:49.795 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:19:49.795 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:19:49.795 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:19:49.797 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:19:49.798 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.086 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:19:49.798 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:20:09.799 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:20:09.836 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:20:09.836 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:20:09.836 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:20:09.836 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:20:09.836 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:20:09.836 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:20:09.836 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:20:09.838 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:20:09.838 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.127 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:20:09.839 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:20:09.839 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:20:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:20:09.839 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:20:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:20:09.839 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:20:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:20:09.839 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:20:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:20:09.839 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:20:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:20:09.839 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:20:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:20:09.839 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:20:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:20:09.839 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:20:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:20:09.839 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:20:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:20:09.839 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:20:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:20:09.839 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:20:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:20:09.839 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:20:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:20:09.839 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:20:09.839 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:20:09.839 139328 ERROR oslo_service.periodic_task >2018-06-24 22:20:39.700 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:20:39.701 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:20:39.708 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:20:39.709 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:20:39.741 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:20:39.741 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:20:39.741 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:20:39.741 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:20:39.741 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:20:39.741 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:20:39.741 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:20:39.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:20:39.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:20:39.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:20:49.745 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:20:49.778 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:20:49.778 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:20:49.778 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:20:49.778 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:20:49.778 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:20:49.778 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:20:49.778 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:20:49.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:20:49.781 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:20:49.781 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:21:09.781 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:21:09.813 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:21:09.813 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:21:09.813 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:21:09.813 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:21:09.813 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:21:09.813 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:21:09.813 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:21:09.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:21:09.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.107 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:21:09.816 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:21:09.816 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:21:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:21:09.816 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:21:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:21:09.816 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:21:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:21:09.816 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:21:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:21:09.816 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:21:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:21:09.816 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:21:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:21:09.816 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:21:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:21:09.816 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:21:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:21:09.816 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:21:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:21:09.816 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:21:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:21:09.816 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:21:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:21:09.816 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:21:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:21:09.816 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:21:09.816 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:21:09.816 139328 ERROR oslo_service.periodic_task >2018-06-24 22:21:39.697 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:21:39.698 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:21:39.704 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:21:39.705 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:21:39.738 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:21:39.738 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:21:39.738 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:21:39.738 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:21:39.738 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:21:39.738 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:21:39.738 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:21:39.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:21:39.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:21:39.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:21:49.741 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:21:49.778 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:21:49.778 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:21:49.778 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:21:49.778 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:21:49.778 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:21:49.778 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:21:49.778 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:21:49.781 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:21:49.781 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:21:49.781 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:22:09.782 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:22:09.815 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:22:09.815 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:22:09.815 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:22:09.815 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:22:09.815 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:22:09.815 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:22:09.815 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:22:09.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:22:09.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:22:09.818 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:22:09.818 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:22:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:22:09.818 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:22:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:22:09.818 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:22:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:22:09.818 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:22:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:22:09.818 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:22:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:22:09.818 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:22:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:22:09.818 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:22:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:22:09.818 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:22:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:22:09.818 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:22:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:22:09.818 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:22:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:22:09.818 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:22:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:22:09.818 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:22:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:22:09.818 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:22:09.818 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:22:09.818 139328 ERROR oslo_service.periodic_task >2018-06-24 22:22:39.699 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:22:39.700 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:22:39.707 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:22:39.707 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:22:39.741 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:22:39.741 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:22:39.741 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:22:39.741 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:22:39.741 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:22:39.741 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:22:39.741 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:22:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:22:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:22:39.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:22:49.744 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:22:49.777 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:22:49.777 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:22:49.777 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:22:49.777 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:22:49.777 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:22:49.777 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:22:49.777 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:22:49.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:22:49.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:22:49.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:23:09.780 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:23:09.812 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:23:09.812 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:23:09.812 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:23:09.812 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:23:09.812 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:23:09.812 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:23:09.812 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:23:09.814 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:23:09.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.106 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:23:09.815 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:23:09.815 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:23:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:23:09.815 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:23:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:23:09.815 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:23:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:23:09.815 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:23:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:23:09.815 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:23:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:23:09.815 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:23:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:23:09.815 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:23:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:23:09.815 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:23:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:23:09.815 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:23:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:23:09.815 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:23:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:23:09.815 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:23:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:23:09.815 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:23:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:23:09.815 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:23:09.815 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:23:09.815 139328 ERROR oslo_service.periodic_task >2018-06-24 22:23:39.696 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:23:39.697 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:23:39.705 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:23:39.705 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:23:39.743 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:23:39.743 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:23:39.743 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:23:39.743 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:23:39.743 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:23:39.743 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:23:39.743 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:23:39.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:23:39.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:23:39.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:23:49.747 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:23:49.782 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:23:49.782 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:23:49.782 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:23:49.782 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:23:49.782 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:23:49.782 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:23:49.782 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:23:49.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:23:49.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:23:49.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:24:09.786 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:24:09.819 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:24:09.819 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:24:09.819 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:24:09.819 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:24:09.819 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:24:09.819 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:24:09.819 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:24:09.821 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:24:09.822 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:24:09.822 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:24:09.822 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:24:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:24:09.822 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:24:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:24:09.822 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:24:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:24:09.822 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:24:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:24:09.822 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:24:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:24:09.822 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:24:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:24:09.822 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:24:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:24:09.822 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:24:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:24:09.822 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:24:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:24:09.822 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:24:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:24:09.822 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:24:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:24:09.822 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:24:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:24:09.822 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:24:09.822 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:24:09.822 139328 ERROR oslo_service.periodic_task >2018-06-24 22:24:39.693 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:24:39.694 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:24:39.701 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:24:39.702 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:24:39.736 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:24:39.736 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:24:39.736 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:24:39.736 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:24:39.736 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:24:39.736 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:24:39.736 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:24:39.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:24:39.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:24:39.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:24:49.740 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:24:49.773 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:24:49.773 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:24:49.773 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:24:49.773 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:24:49.773 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:24:49.773 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:24:49.773 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:24:49.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:24:49.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:24:49.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:25:09.776 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:25:09.809 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:25:09.809 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:25:09.809 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:25:09.809 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:25:09.809 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:25:09.809 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:25:09.809 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:25:09.811 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:25:09.812 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:25:09.812 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:25:09.812 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:25:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:25:09.812 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:25:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:25:09.812 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:25:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:25:09.812 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:25:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:25:09.812 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:25:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:25:09.812 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:25:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:25:09.812 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:25:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:25:09.812 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:25:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:25:09.812 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:25:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:25:09.812 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:25:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:25:09.812 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:25:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:25:09.812 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:25:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:25:09.812 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:25:09.812 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:25:09.812 139328 ERROR oslo_service.periodic_task >2018-06-24 22:25:39.693 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:25:39.694 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:25:39.701 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:25:39.702 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:25:39.740 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:25:39.740 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:25:39.740 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:25:39.740 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:25:39.740 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:25:39.740 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:25:39.740 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:25:39.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:25:39.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:25:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:25:49.743 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:25:49.780 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:25:49.780 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:25:49.780 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:25:49.780 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:25:49.780 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:25:49.780 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:25:49.780 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:25:49.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:25:49.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.08 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:25:49.783 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:26:09.783 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:26:09.819 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:26:09.819 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:26:09.819 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:26:09.819 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:26:09.819 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:26:09.819 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:26:09.819 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:26:09.822 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:26:09.822 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.12 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:26:09.822 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:26:09.822 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:26:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:26:09.822 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:26:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:26:09.822 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:26:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:26:09.822 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:26:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:26:09.822 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:26:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:26:09.822 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:26:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:26:09.822 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:26:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:26:09.822 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:26:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:26:09.822 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:26:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:26:09.822 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:26:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:26:09.822 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:26:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:26:09.822 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:26:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:26:09.822 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:26:09.822 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:26:09.822 139328 ERROR oslo_service.periodic_task >2018-06-24 22:26:39.694 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:26:39.695 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:26:39.704 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:26:39.704 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:26:39.738 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:26:39.738 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:26:39.738 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:26:39.738 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:26:39.738 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:26:39.738 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:26:39.738 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:26:39.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:26:39.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:26:39.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:26:49.742 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:26:49.776 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:26:49.776 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:26:49.776 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:26:49.776 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:26:49.776 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:26:49.776 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:26:49.776 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:26:49.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:26:49.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:26:49.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:27:09.780 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:27:09.813 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:27:09.813 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:27:09.813 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:27:09.813 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:27:09.813 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:27:09.813 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:27:09.813 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:27:09.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:27:09.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:27:09.816 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:27:09.816 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:27:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:27:09.816 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:27:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:27:09.816 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:27:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:27:09.816 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:27:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:27:09.816 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:27:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:27:09.816 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:27:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:27:09.816 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:27:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:27:09.816 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:27:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:27:09.816 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:27:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:27:09.816 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:27:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:27:09.816 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:27:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:27:09.816 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:27:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:27:09.816 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:27:09.816 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:27:09.816 139328 ERROR oslo_service.periodic_task >2018-06-24 22:27:39.697 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:27:39.698 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:27:39.707 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:27:39.708 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:27:39.742 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:27:39.742 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:27:39.742 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:27:39.742 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:27:39.742 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:27:39.742 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:27:39.742 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:27:39.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:27:39.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:27:39.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:27:49.745 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:27:49.779 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:27:49.779 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:27:49.779 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:27:49.779 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:27:49.779 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:27:49.779 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:27:49.779 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:27:49.781 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:27:49.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:27:49.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:28:09.783 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:28:09.820 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:28:09.820 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:28:09.820 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:28:09.820 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:28:09.820 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:28:09.820 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:28:09.820 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:28:09.822 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:28:09.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:28:09.823 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:28:09.823 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:28:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:28:09.823 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:28:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:28:09.823 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:28:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:28:09.823 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:28:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:28:09.823 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:28:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:28:09.823 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:28:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:28:09.823 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:28:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:28:09.823 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:28:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:28:09.823 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:28:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:28:09.823 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:28:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:28:09.823 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:28:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:28:09.823 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:28:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:28:09.823 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:28:09.823 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:28:09.823 139328 ERROR oslo_service.periodic_task >2018-06-24 22:28:39.695 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:28:39.696 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:28:39.702 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:28:39.703 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:28:39.736 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:28:39.736 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:28:39.736 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:28:39.736 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:28:39.736 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:28:39.736 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:28:39.736 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:28:39.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:28:39.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:28:39.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:28:49.741 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:28:49.774 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:28:49.774 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:28:49.774 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:28:49.774 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:28:49.774 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:28:49.774 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:28:49.774 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:28:49.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:28:49.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:28:49.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:29:09.777 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:29:09.811 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:29:09.811 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:29:09.811 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:29:09.811 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:29:09.811 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:29:09.811 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:29:09.811 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:29:09.814 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:29:09.814 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:29:09.815 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:29:09.815 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:29:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:29:09.815 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:29:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:29:09.815 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:29:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:29:09.815 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:29:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:29:09.815 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:29:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:29:09.815 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:29:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:29:09.815 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:29:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:29:09.815 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:29:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:29:09.815 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:29:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:29:09.815 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:29:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:29:09.815 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:29:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:29:09.815 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:29:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:29:09.815 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:29:09.815 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:29:09.815 139328 ERROR oslo_service.periodic_task >2018-06-24 22:29:39.696 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:29:39.698 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:29:39.705 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:29:39.705 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:29:39.738 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:29:39.738 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:29:39.738 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:29:39.738 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:29:39.738 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:29:39.738 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:29:39.738 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:29:39.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:29:39.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:29:39.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:29:49.742 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:29:49.782 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:29:49.782 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:29:49.782 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:29:49.782 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:29:49.782 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:29:49.782 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:29:49.782 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:29:49.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:29:49.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:29:49.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:30:09.786 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:30:09.819 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:30:09.819 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:30:09.819 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:30:09.819 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:30:09.819 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:30:09.819 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:30:09.819 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:30:09.821 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:30:09.822 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:30:09.822 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:30:09.822 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:30:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:30:09.822 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:30:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:30:09.822 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:30:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:30:09.822 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:30:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:30:09.822 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:30:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:30:09.822 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:30:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:30:09.822 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:30:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:30:09.822 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:30:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:30:09.822 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:30:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:30:09.822 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:30:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:30:09.822 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:30:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:30:09.822 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:30:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:30:09.822 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:30:09.822 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:30:09.822 139328 ERROR oslo_service.periodic_task >2018-06-24 22:30:39.694 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:30:39.694 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:30:39.701 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:30:39.702 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:30:39.741 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:30:39.741 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:30:39.741 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:30:39.741 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:30:39.741 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:30:39.741 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:30:39.741 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:30:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:30:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.042 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:30:39.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:30:49.744 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:30:49.777 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:30:49.777 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:30:49.777 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:30:49.777 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:30:49.777 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:30:49.777 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:30:49.777 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:30:49.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:30:49.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:30:49.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:31:09.780 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:31:09.815 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:31:09.815 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:31:09.815 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:31:09.815 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:31:09.815 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:31:09.815 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:31:09.815 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:31:09.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:31:09.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:31:09.817 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:31:09.817 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:31:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:31:09.817 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:31:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:31:09.817 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:31:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:31:09.817 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:31:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:31:09.817 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:31:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:31:09.817 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:31:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:31:09.817 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:31:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:31:09.817 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:31:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:31:09.817 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:31:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:31:09.817 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:31:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:31:09.817 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:31:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:31:09.817 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:31:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:31:09.817 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:31:09.817 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:31:09.817 139328 ERROR oslo_service.periodic_task >2018-06-24 22:31:39.689 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:31:39.690 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:31:39.697 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:31:39.697 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:31:39.747 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:31:39.747 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:31:39.747 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:31:39.747 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:31:39.747 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:31:39.747 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:31:39.747 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:31:39.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:31:39.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.051 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:31:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:31:49.750 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:31:49.783 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:31:49.783 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:31:49.783 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:31:49.783 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:31:49.783 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:31:49.783 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:31:49.783 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:31:49.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:31:49.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.088 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:31:49.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:32:09.787 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:32:09.821 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:32:09.821 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:32:09.821 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:32:09.821 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:32:09.821 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:32:09.821 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:32:09.821 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:32:09.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:32:09.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.126 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:32:09.824 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:32:09.824 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:32:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:32:09.824 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:32:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:32:09.824 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:32:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:32:09.824 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:32:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:32:09.824 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:32:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:32:09.824 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:32:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:32:09.824 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:32:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:32:09.824 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:32:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:32:09.824 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:32:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:32:09.824 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:32:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:32:09.824 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:32:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:32:09.824 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:32:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:32:09.824 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:32:09.824 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:32:09.824 139328 ERROR oslo_service.periodic_task >2018-06-24 22:32:39.686 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:32:39.687 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:32:39.693 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:32:39.694 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:32:39.728 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:32:39.728 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:32:39.728 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:32:39.728 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:32:39.728 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:32:39.728 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:32:39.728 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:32:39.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:32:39.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:32:39.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:32:49.732 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:32:49.766 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:32:49.766 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:32:49.766 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:32:49.766 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:32:49.766 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:32:49.766 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:32:49.766 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:32:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:32:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:32:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:33:09.769 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:33:09.808 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:33:09.808 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:33:09.808 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:33:09.808 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:33:09.808 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:33:09.808 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:33:09.808 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:33:09.810 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:33:09.810 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:33:09.811 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:33:09.811 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:33:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:33:09.811 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:33:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:33:09.811 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:33:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:33:09.811 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:33:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:33:09.811 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:33:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:33:09.811 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:33:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:33:09.811 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:33:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:33:09.811 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:33:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:33:09.811 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:33:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:33:09.811 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:33:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:33:09.811 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:33:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:33:09.811 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:33:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:33:09.811 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:33:09.811 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:33:09.811 139328 ERROR oslo_service.periodic_task >2018-06-24 22:33:39.683 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:33:39.684 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:33:39.695 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:33:39.696 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:33:39.730 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:33:39.730 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:33:39.730 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:33:39.730 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:33:39.730 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:33:39.730 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:33:39.730 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:33:39.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:33:39.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:33:39.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:33:49.734 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:33:49.766 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:33:49.766 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:33:49.766 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:33:49.766 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:33:49.766 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:33:49.766 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:33:49.766 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:33:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:33:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:33:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:34:09.769 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:34:09.804 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:34:09.804 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:34:09.804 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:34:09.804 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:34:09.804 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:34:09.804 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:34:09.804 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:34:09.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:34:09.807 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:34:09.807 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:34:09.807 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:34:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:34:09.807 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:34:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:34:09.807 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:34:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:34:09.807 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:34:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:34:09.807 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:34:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:34:09.807 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:34:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:34:09.807 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:34:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:34:09.807 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:34:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:34:09.807 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:34:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:34:09.807 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:34:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:34:09.807 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:34:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:34:09.807 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:34:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:34:09.807 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:34:09.807 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:34:09.807 139328 ERROR oslo_service.periodic_task >2018-06-24 22:34:39.679 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:34:39.680 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:35:39.678 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:35:39.679 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:35:39.686 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:35:39.686 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:35:39.721 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:35:39.721 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:35:39.721 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:35:39.721 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:35:39.721 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:35:39.721 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:35:39.721 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:35:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:35:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:35:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:35:49.725 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:35:49.760 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:35:49.760 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:35:49.760 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:35:49.760 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:35:49.760 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:35:49.760 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:35:49.760 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:35:49.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:35:49.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:35:49.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:36:09.764 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:36:09.799 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:36:09.799 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:36:09.799 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:36:09.799 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:36:09.799 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:36:09.799 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:36:09.799 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:36:09.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:36:09.803 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:36:09.803 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:36:09.803 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:36:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:36:09.803 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:36:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:36:09.803 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:36:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:36:09.803 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:36:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:36:09.803 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:36:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:36:09.803 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:36:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:36:09.803 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:36:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:36:09.803 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:36:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:36:09.803 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:36:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:36:09.803 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:36:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:36:09.803 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:36:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:36:09.803 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:36:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:36:09.803 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:36:09.803 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:36:09.803 139328 ERROR oslo_service.periodic_task >2018-06-24 22:36:39.675 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:36:39.676 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:36:39.686 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:36:39.687 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:36:39.724 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:36:39.724 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:36:39.724 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:36:39.724 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:36:39.724 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:36:39.724 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:36:39.724 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:36:39.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:36:39.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:36:39.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:36:49.728 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:36:49.763 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:36:49.763 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:36:49.763 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:36:49.763 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:36:49.763 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:36:49.763 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:36:49.763 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:36:49.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:36:49.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:36:49.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:37:09.767 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:37:09.803 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:37:09.803 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:37:09.803 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:37:09.803 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:37:09.803 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:37:09.803 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:37:09.803 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:37:09.805 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:37:09.805 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:37:09.805 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:37:09.805 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:37:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:37:09.805 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:37:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:37:09.805 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:37:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:37:09.805 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:37:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:37:09.805 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:37:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:37:09.805 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:37:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:37:09.805 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:37:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:37:09.805 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:37:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:37:09.805 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:37:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:37:09.805 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:37:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:37:09.805 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:37:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:37:09.805 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:37:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:37:09.805 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:37:09.805 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:37:09.805 139328 ERROR oslo_service.periodic_task >2018-06-24 22:37:39.677 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:37:39.678 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:37:39.687 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:37:39.688 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:37:39.725 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:37:39.725 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:37:39.725 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:37:39.725 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:37:39.725 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:37:39.725 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:37:39.725 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:37:39.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:37:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:37:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:37:49.729 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:37:49.770 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:37:49.770 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:37:49.770 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:37:49.770 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:37:49.770 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:37:49.770 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:37:49.770 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:37:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:37:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.086 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:37:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:38:09.775 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:38:09.809 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:38:09.809 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:38:09.809 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:38:09.809 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:38:09.809 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:38:09.809 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:38:09.809 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:38:09.811 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:38:09.812 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.124 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:38:09.812 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:38:09.812 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:38:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:38:09.812 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:38:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:38:09.812 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:38:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:38:09.812 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:38:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:38:09.812 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:38:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:38:09.812 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:38:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:38:09.812 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:38:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:38:09.812 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:38:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:38:09.812 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:38:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:38:09.812 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:38:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:38:09.812 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:38:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:38:09.812 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:38:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:38:09.812 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:38:09.812 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:38:09.812 139328 ERROR oslo_service.periodic_task >2018-06-24 22:38:39.673 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:38:39.674 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:38:39.681 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:38:39.682 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:38:39.716 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:38:39.716 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:38:39.716 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:38:39.716 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:38:39.716 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:38:39.716 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:38:39.716 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:38:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:38:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:38:39.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:38:49.719 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:38:49.753 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:38:49.753 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:38:49.753 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:38:49.753 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:38:49.753 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:38:49.753 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:38:49.753 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:38:49.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:38:49.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:38:49.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:39:09.757 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:39:09.795 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:39:09.795 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:39:09.795 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:39:09.795 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:39:09.795 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:39:09.795 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:39:09.795 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:39:09.798 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:39:09.799 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:39:09.799 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:39:09.799 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:39:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:39:09.799 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:39:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:39:09.799 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:39:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:39:09.799 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:39:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:39:09.799 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:39:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:39:09.799 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:39:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:39:09.799 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:39:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:39:09.799 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:39:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:39:09.799 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:39:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:39:09.799 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:39:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:39:09.799 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:39:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:39:09.799 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:39:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:39:09.799 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:39:09.799 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:39:09.799 139328 ERROR oslo_service.periodic_task >2018-06-24 22:39:39.671 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:39:39.672 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:39:39.679 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:39:39.680 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:39:39.712 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:39:39.712 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:39:39.712 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:39:39.712 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:39:39.712 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:39:39.712 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:39:39.712 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:39:39.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:39:39.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:39:39.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:39:49.715 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:39:49.749 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:39:49.749 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:39:49.749 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:39:49.749 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:39:49.749 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:39:49.749 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:39:49.749 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:39:49.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:39:49.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:39:49.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:40:09.752 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:40:09.786 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:40:09.786 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:40:09.786 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:40:09.786 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:40:09.786 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:40:09.786 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:40:09.786 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:40:09.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:40:09.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:40:09.789 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:40:09.789 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:40:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:40:09.789 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:40:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:40:09.789 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:40:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:40:09.789 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:40:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:40:09.789 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:40:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:40:09.789 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:40:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:40:09.789 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:40:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:40:09.789 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:40:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:40:09.789 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:40:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:40:09.789 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:40:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:40:09.789 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:40:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:40:09.789 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:40:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:40:09.789 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:40:09.789 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:40:09.789 139328 ERROR oslo_service.periodic_task >2018-06-24 22:40:39.670 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:40:39.671 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:40:39.678 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:40:39.678 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:40:39.712 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:40:39.712 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:40:39.712 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:40:39.712 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:40:39.712 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:40:39.712 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:40:39.712 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:40:39.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:40:39.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:40:39.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:40:49.716 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:40:49.749 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:40:49.749 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:40:49.749 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:40:49.749 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:40:49.749 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:40:49.749 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:40:49.749 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:40:49.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:40:49.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:40:49.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:41:09.752 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:41:09.786 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:41:09.786 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:41:09.786 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:41:09.786 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:41:09.786 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:41:09.786 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:41:09.786 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:41:09.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:41:09.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:41:09.789 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:41:09.789 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:41:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:41:09.789 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:41:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:41:09.789 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:41:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:41:09.789 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:41:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:41:09.789 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:41:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:41:09.789 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:41:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:41:09.789 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:41:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:41:09.789 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:41:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:41:09.789 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:41:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:41:09.789 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:41:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:41:09.789 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:41:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:41:09.789 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:41:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:41:09.789 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:41:09.789 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:41:09.789 139328 ERROR oslo_service.periodic_task >2018-06-24 22:41:39.671 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:41:39.672 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:41:39.679 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:41:39.680 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:41:39.714 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:41:39.714 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:41:39.714 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:41:39.714 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:41:39.714 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:41:39.714 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:41:39.714 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:41:39.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:41:39.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:41:39.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:41:49.717 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:41:49.750 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:41:49.750 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:41:49.750 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:41:49.750 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:41:49.750 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:41:49.750 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:41:49.750 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:41:49.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:41:49.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:41:49.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:42:09.753 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:42:09.787 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:42:09.787 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:42:09.787 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:42:09.787 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:42:09.787 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:42:09.787 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:42:09.787 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:42:09.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:42:09.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:42:09.790 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:42:09.790 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:42:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:42:09.790 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:42:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:42:09.790 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:42:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:42:09.790 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:42:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:42:09.790 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:42:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:42:09.790 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:42:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:42:09.790 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:42:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:42:09.790 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:42:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:42:09.790 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:42:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:42:09.790 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:42:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:42:09.790 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:42:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:42:09.790 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:42:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:42:09.790 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:42:09.790 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:42:09.790 139328 ERROR oslo_service.periodic_task >2018-06-24 22:42:39.672 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:42:39.673 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:42:39.680 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:42:39.681 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:42:39.717 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:42:39.717 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:42:39.717 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:42:39.717 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:42:39.717 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:42:39.717 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:42:39.717 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:42:39.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:42:39.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:42:39.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:42:49.721 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:42:49.757 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:42:49.757 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:42:49.757 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:42:49.757 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:42:49.757 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:42:49.757 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:42:49.757 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:42:49.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:42:49.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:42:49.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:43:09.760 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:43:09.796 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:43:09.796 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:43:09.796 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:43:09.796 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:43:09.796 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:43:09.796 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:43:09.796 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:43:09.798 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:43:09.798 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:43:09.799 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:43:09.799 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:43:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:43:09.799 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:43:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:43:09.799 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:43:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:43:09.799 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:43:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:43:09.799 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:43:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:43:09.799 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:43:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:43:09.799 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:43:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:43:09.799 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:43:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:43:09.799 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:43:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:43:09.799 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:43:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:43:09.799 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:43:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:43:09.799 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:43:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:43:09.799 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:43:09.799 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:43:09.799 139328 ERROR oslo_service.periodic_task >2018-06-24 22:43:39.670 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:43:39.671 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:43:39.678 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:43:39.678 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:43:39.711 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:43:39.711 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:43:39.711 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:43:39.711 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:43:39.711 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:43:39.711 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:43:39.711 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:43:39.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:43:39.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:43:39.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:43:49.715 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:43:49.749 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:43:49.749 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:43:49.749 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:43:49.749 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:43:49.749 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:43:49.749 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:43:49.749 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:43:49.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:43:49.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:43:49.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:44:09.753 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:44:09.786 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:44:09.786 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:44:09.786 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:44:09.786 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:44:09.786 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:44:09.786 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:44:09.786 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:44:09.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:44:09.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:44:09.789 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:44:09.789 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:44:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:44:09.789 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:44:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:44:09.789 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:44:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:44:09.789 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:44:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:44:09.789 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:44:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:44:09.789 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:44:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:44:09.789 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:44:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:44:09.789 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:44:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:44:09.789 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:44:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:44:09.789 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:44:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:44:09.789 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:44:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:44:09.789 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:44:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:44:09.789 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:44:09.789 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:44:09.789 139328 ERROR oslo_service.periodic_task >2018-06-24 22:44:39.670 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:44:39.671 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:44:39.679 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:44:39.679 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:44:39.715 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:44:39.715 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:44:39.715 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:44:39.715 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:44:39.715 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:44:39.715 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:44:39.715 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:44:39.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:44:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:44:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:44:49.719 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:44:49.751 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:44:49.751 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:44:49.751 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:44:49.751 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:44:49.751 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:44:49.751 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:44:49.751 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:44:49.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:44:49.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:44:49.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:45:09.754 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:45:09.787 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:45:09.787 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:45:09.787 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:45:09.787 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:45:09.787 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:45:09.787 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:45:09.787 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:45:09.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:45:09.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:45:09.790 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:45:09.790 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:45:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:45:09.790 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:45:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:45:09.790 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:45:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:45:09.790 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:45:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:45:09.790 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:45:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:45:09.790 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:45:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:45:09.790 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:45:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:45:09.790 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:45:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:45:09.790 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:45:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:45:09.790 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:45:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:45:09.790 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:45:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:45:09.790 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:45:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:45:09.790 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:45:09.790 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:45:09.790 139328 ERROR oslo_service.periodic_task >2018-06-24 22:45:39.671 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:45:39.672 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:45:39.680 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:45:39.680 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:45:39.714 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:45:39.714 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:45:39.714 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:45:39.714 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:45:39.714 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:45:39.714 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:45:39.714 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:45:39.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:45:39.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:45:39.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:45:49.718 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:45:49.763 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:45:49.763 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:45:49.763 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:45:49.763 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:45:49.763 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:45:49.763 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:45:49.763 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:45:49.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:45:49.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.085 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:45:49.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:46:09.767 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:46:09.800 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:46:09.800 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:46:09.800 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:46:09.800 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:46:09.800 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:46:09.800 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:46:09.800 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:46:09.803 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:46:09.803 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.122 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:46:09.804 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:46:09.804 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:46:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:46:09.804 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:46:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:46:09.804 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:46:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:46:09.804 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:46:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:46:09.804 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:46:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:46:09.804 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:46:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:46:09.804 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:46:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:46:09.804 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:46:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:46:09.804 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:46:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:46:09.804 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:46:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:46:09.804 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:46:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:46:09.804 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:46:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:46:09.804 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:46:09.804 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:46:09.804 139328 ERROR oslo_service.periodic_task >2018-06-24 22:46:39.675 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:46:39.676 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:46:39.683 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:46:39.683 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:46:39.718 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:46:39.718 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:46:39.718 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:46:39.718 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:46:39.718 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:46:39.718 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:46:39.718 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:46:39.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:46:39.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:46:39.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:46:49.722 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:46:49.758 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:46:49.758 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:46:49.758 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:46:49.758 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:46:49.758 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:46:49.758 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:46:49.758 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:46:49.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:46:49.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:46:49.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:47:09.762 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:47:09.796 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:47:09.796 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:47:09.796 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:47:09.796 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:47:09.796 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:47:09.796 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:47:09.796 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:47:09.798 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:47:09.798 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:47:09.799 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:47:09.799 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:47:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:47:09.799 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:47:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:47:09.799 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:47:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:47:09.799 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:47:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:47:09.799 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:47:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:47:09.799 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:47:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:47:09.799 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:47:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:47:09.799 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:47:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:47:09.799 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:47:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:47:09.799 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:47:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:47:09.799 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:47:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:47:09.799 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:47:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:47:09.799 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:47:09.799 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:47:09.799 139328 ERROR oslo_service.periodic_task >2018-06-24 22:47:39.680 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:47:39.681 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:47:39.688 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:47:39.689 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:47:39.739 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:47:39.739 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:47:39.739 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:47:39.739 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:47:39.739 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:47:39.739 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:47:39.739 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:47:39.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:47:39.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.053 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:47:39.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:47:49.742 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:47:49.775 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:47:49.775 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:47:49.775 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:47:49.775 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:47:49.775 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:47:49.775 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:47:49.775 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:47:49.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:47:49.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.089 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:47:49.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:48:09.779 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:48:09.812 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:48:09.812 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:48:09.812 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:48:09.812 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:48:09.812 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:48:09.812 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:48:09.812 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:48:09.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:48:09.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.126 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:48:09.815 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:48:09.815 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:48:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:48:09.815 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:48:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:48:09.815 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:48:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:48:09.815 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:48:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:48:09.815 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:48:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:48:09.815 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:48:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:48:09.815 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:48:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:48:09.815 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:48:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:48:09.815 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:48:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:48:09.815 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:48:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:48:09.815 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:48:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:48:09.815 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:48:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:48:09.815 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:48:09.815 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:48:09.815 139328 ERROR oslo_service.periodic_task >2018-06-24 22:48:39.677 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:48:39.678 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:48:39.693 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:48:39.694 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:48:39.729 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:48:39.729 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:48:39.729 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:48:39.729 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:48:39.729 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:48:39.729 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:48:39.729 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:48:39.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:48:39.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:48:39.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:48:49.733 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:48:49.770 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:48:49.770 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:48:49.770 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:48:49.770 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:48:49.770 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:48:49.770 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:48:49.770 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:48:49.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:48:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:48:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:49:09.775 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:49:09.812 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:49:09.812 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:49:09.812 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:49:09.812 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:49:09.812 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:49:09.812 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:49:09.812 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:49:09.814 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:49:09.814 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.121 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:49:09.815 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:49:09.815 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:49:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:49:09.815 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:49:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:49:09.815 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:49:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:49:09.815 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:49:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:49:09.815 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:49:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:49:09.815 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:49:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:49:09.815 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:49:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:49:09.815 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:49:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:49:09.815 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:49:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:49:09.815 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:49:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:49:09.815 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:49:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:49:09.815 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:49:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:49:09.815 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:49:09.815 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:49:09.815 139328 ERROR oslo_service.periodic_task >2018-06-24 22:49:39.676 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:49:39.677 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:49:39.684 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:49:39.685 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:49:39.724 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:49:39.724 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:49:39.724 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:49:39.724 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:49:39.724 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:49:39.724 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:49:39.724 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:49:39.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:49:39.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.042 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:49:39.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:49:49.728 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:49:49.765 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:49:49.765 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:49:49.765 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:49:49.765 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:49:49.765 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:49:49.765 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:49:49.765 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:49:49.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:49:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.083 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:49:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:50:09.768 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:50:09.806 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:50:09.806 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:50:09.806 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:50:09.806 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:50:09.806 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:50:09.806 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:50:09.806 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:50:09.808 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:50:09.808 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.123 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:50:09.809 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:50:09.809 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:50:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:50:09.809 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:50:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:50:09.809 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:50:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:50:09.809 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:50:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:50:09.809 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:50:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:50:09.809 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:50:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:50:09.809 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:50:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:50:09.809 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:50:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:50:09.809 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:50:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:50:09.809 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:50:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:50:09.809 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:50:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:50:09.809 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:50:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:50:09.809 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:50:09.809 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:50:09.809 139328 ERROR oslo_service.periodic_task >2018-06-24 22:50:39.680 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:50:39.681 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:50:39.694 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:50:39.695 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:50:39.729 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:50:39.729 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:50:39.729 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:50:39.729 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:50:39.729 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:50:39.729 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:50:39.729 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:50:39.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:50:39.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:50:39.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:50:49.733 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:50:49.767 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:50:49.767 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:50:49.767 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:50:49.767 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:50:49.767 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:50:49.767 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:50:49.767 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:50:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:50:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:50:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:51:09.770 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:51:09.813 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:51:09.813 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:51:09.813 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:51:09.813 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:51:09.813 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:51:09.813 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:51:09.813 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:51:09.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:51:09.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.122 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:51:09.817 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:51:09.817 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:51:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:51:09.817 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:51:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:51:09.817 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:51:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:51:09.817 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:51:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:51:09.817 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:51:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:51:09.817 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:51:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:51:09.817 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:51:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:51:09.817 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:51:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:51:09.817 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:51:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:51:09.817 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:51:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:51:09.817 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:51:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:51:09.817 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:51:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:51:09.817 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:51:09.817 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:51:09.817 139328 ERROR oslo_service.periodic_task >2018-06-24 22:51:39.679 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:51:39.680 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:51:39.692 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:51:39.693 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:51:39.728 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:51:39.728 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:51:39.728 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:51:39.728 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:51:39.728 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:51:39.728 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:51:39.728 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:51:39.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:51:39.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:51:39.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:51:49.731 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:51:49.763 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:51:49.763 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:51:49.763 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:51:49.763 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:51:49.763 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:51:49.763 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:51:49.763 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:51:49.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:51:49.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:51:49.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:52:09.767 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:52:09.804 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:52:09.804 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:52:09.804 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:52:09.804 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:52:09.804 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:52:09.804 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:52:09.804 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:52:09.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:52:09.807 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:52:09.807 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:52:09.807 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:52:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:52:09.807 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:52:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:52:09.807 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:52:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:52:09.807 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:52:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:52:09.807 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:52:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:52:09.807 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:52:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:52:09.807 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:52:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:52:09.807 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:52:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:52:09.807 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:52:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:52:09.807 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:52:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:52:09.807 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:52:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:52:09.807 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:52:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:52:09.807 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:52:09.807 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:52:09.807 139328 ERROR oslo_service.periodic_task >2018-06-24 22:52:39.678 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:52:39.679 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:52:39.686 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:52:39.687 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:52:39.723 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:52:39.723 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:52:39.723 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:52:39.723 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:52:39.723 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:52:39.723 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:52:39.723 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:52:39.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:52:39.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:52:39.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:52:49.727 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:52:49.761 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:52:49.761 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:52:49.761 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:52:49.761 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:52:49.761 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:52:49.761 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:52:49.761 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:52:49.764 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:52:49.764 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:52:49.764 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:53:09.765 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:53:09.800 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:53:09.800 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:53:09.800 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:53:09.800 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:53:09.800 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:53:09.800 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:53:09.800 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:53:09.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:53:09.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:53:09.803 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:53:09.803 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:53:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:53:09.803 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:53:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:53:09.803 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:53:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:53:09.803 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:53:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:53:09.803 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:53:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:53:09.803 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:53:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:53:09.803 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:53:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:53:09.803 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:53:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:53:09.803 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:53:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:53:09.803 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:53:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:53:09.803 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:53:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:53:09.803 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:53:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:53:09.803 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:53:09.803 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:53:09.803 139328 ERROR oslo_service.periodic_task >2018-06-24 22:53:39.674 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:53:39.675 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:53:39.682 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:53:39.682 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:53:39.715 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:53:39.715 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:53:39.715 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:53:39.715 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:53:39.715 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:53:39.715 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:53:39.715 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:53:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:53:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:53:39.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:53:49.719 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:53:49.753 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:53:49.753 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:53:49.753 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:53:49.753 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:53:49.753 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:53:49.753 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:53:49.753 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:53:49.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:53:49.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:53:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:54:09.757 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:54:09.796 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:54:09.796 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:54:09.796 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:54:09.796 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:54:09.796 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:54:09.796 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:54:09.796 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:54:09.799 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:54:09.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:54:09.800 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:54:09.800 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:54:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:54:09.800 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:54:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:54:09.800 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:54:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:54:09.800 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:54:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:54:09.800 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:54:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:54:09.800 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:54:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:54:09.800 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:54:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:54:09.800 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:54:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:54:09.800 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:54:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:54:09.800 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:54:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:54:09.800 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:54:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:54:09.800 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:54:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:54:09.800 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:54:09.800 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:54:09.800 139328 ERROR oslo_service.periodic_task >2018-06-24 22:54:39.672 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:54:39.673 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:54:39.680 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:54:39.681 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:54:39.718 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:54:39.718 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:54:39.718 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:54:39.718 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:54:39.718 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:54:39.718 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:54:39.718 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:54:39.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:54:39.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:54:39.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:54:49.722 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:54:49.758 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:54:49.758 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:54:49.758 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:54:49.758 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:54:49.758 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:54:49.758 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:54:49.758 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:54:49.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:54:49.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.08 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:54:49.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:55:09.761 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:55:09.796 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:55:09.796 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:55:09.796 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:55:09.796 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:55:09.796 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:55:09.796 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:55:09.796 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:55:09.798 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:55:09.799 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.118 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:55:09.799 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:55:09.799 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:55:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:55:09.799 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:55:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:55:09.799 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:55:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:55:09.799 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:55:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:55:09.799 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:55:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:55:09.799 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:55:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:55:09.799 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:55:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:55:09.799 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:55:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:55:09.799 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:55:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:55:09.799 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:55:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:55:09.799 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:55:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:55:09.799 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:55:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:55:09.799 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:55:09.799 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:55:09.799 139328 ERROR oslo_service.periodic_task >2018-06-24 22:55:39.671 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:55:39.672 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:55:39.680 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:55:39.680 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:55:39.715 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:55:39.715 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:55:39.715 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:55:39.715 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:55:39.715 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:55:39.715 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:55:39.715 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:55:39.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:55:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:55:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:55:49.719 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:55:49.752 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:55:49.752 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:55:49.752 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:55:49.752 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:55:49.752 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:55:49.752 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:55:49.752 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:55:49.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:55:49.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:55:49.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:56:09.755 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:56:09.792 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:56:09.792 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:56:09.792 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:56:09.792 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:56:09.792 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:56:09.792 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:56:09.792 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:56:09.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:56:09.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:56:09.794 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:56:09.794 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:56:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:56:09.794 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:56:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:56:09.794 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:56:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:56:09.794 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:56:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:56:09.794 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:56:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:56:09.794 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:56:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:56:09.794 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:56:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:56:09.794 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:56:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:56:09.794 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:56:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:56:09.794 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:56:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:56:09.794 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:56:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:56:09.794 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:56:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:56:09.794 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:56:09.794 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:56:09.794 139328 ERROR oslo_service.periodic_task >2018-06-24 22:56:39.666 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:56:39.667 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:56:39.679 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:56:39.679 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:56:39.714 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:56:39.714 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:56:39.714 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:56:39.714 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:56:39.714 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:56:39.714 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:56:39.714 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:56:39.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:56:39.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:56:39.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:56:49.718 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:56:49.752 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:56:49.752 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:56:49.752 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:56:49.752 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:56:49.752 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:56:49.752 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:56:49.752 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:56:49.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:56:49.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:56:49.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:57:09.755 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:57:09.788 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:57:09.788 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:57:09.788 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:57:09.788 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:57:09.788 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:57:09.788 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:57:09.788 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:57:09.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:57:09.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:57:09.791 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:57:09.791 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:57:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:57:09.791 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:57:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:57:09.791 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:57:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:57:09.791 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:57:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:57:09.791 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:57:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:57:09.791 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:57:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:57:09.791 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:57:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:57:09.791 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:57:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:57:09.791 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:57:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:57:09.791 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:57:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:57:09.791 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:57:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:57:09.791 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:57:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:57:09.791 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:57:09.791 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:57:09.791 139328 ERROR oslo_service.periodic_task >2018-06-24 22:57:39.662 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:57:39.663 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:57:39.698 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:57:39.698 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:57:39.698 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:57:39.698 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:57:39.698 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:57:39.698 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:57:39.698 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:57:39.700 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:57:39.700 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:57:39.700 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:57:49.701 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:57:49.735 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:57:49.735 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:57:49.735 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:57:49.735 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:57:49.735 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:57:49.735 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:57:49.735 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:57:49.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:57:49.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:57:49.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:58:09.738 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:58:09.779 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:58:09.779 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:58:09.779 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:58:09.779 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:58:09.779 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:58:09.779 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:58:09.779 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:58:09.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:58:09.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.118 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:58:09.782 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:58:09.782 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:58:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:58:09.782 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:58:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:58:09.782 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:58:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:58:09.782 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:58:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:58:09.782 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:58:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:58:09.782 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:58:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:58:09.782 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:58:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:58:09.782 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:58:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:58:09.782 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:58:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:58:09.782 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:58:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:58:09.782 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:58:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:58:09.782 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:58:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:58:09.782 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:58:09.782 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:58:09.782 139328 ERROR oslo_service.periodic_task >2018-06-24 22:58:39.664 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:58:39.665 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:58:39.672 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:58:39.672 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:58:39.708 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:58:39.708 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:58:39.708 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:58:39.708 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:58:39.708 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:58:39.708 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:58:39.708 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:58:39.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:58:39.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:58:39.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:58:49.713 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:58:49.766 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:58:49.766 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:58:49.766 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:58:49.766 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:58:49.766 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:58:49.766 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:58:49.766 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:58:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:58:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.096 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:58:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:59:09.770 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:59:09.809 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:59:09.809 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:59:09.809 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:59:09.809 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:59:09.809 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:59:09.809 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:59:09.809 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:59:09.811 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:59:09.811 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.138 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:59:09.812 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:59:09.812 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 22:59:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 22:59:09.812 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 22:59:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 22:59:09.812 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 22:59:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 22:59:09.812 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 22:59:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 22:59:09.812 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 22:59:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 22:59:09.812 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 22:59:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 22:59:09.812 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 22:59:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 22:59:09.812 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 22:59:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 22:59:09.812 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 22:59:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 22:59:09.812 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 22:59:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 22:59:09.812 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 22:59:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 22:59:09.812 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 22:59:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 22:59:09.812 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 22:59:09.812 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 22:59:09.812 139328 ERROR oslo_service.periodic_task >2018-06-24 22:59:39.663 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:59:39.664 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 22:59:39.672 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 22:59:39.673 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:59:39.707 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:59:39.707 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:59:39.707 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:59:39.707 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:59:39.707 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:59:39.707 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:59:39.707 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:59:39.709 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:59:39.709 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:59:39.710 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 22:59:49.710 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 22:59:49.745 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 22:59:49.745 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 22:59:49.745 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 22:59:49.745 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 22:59:49.745 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 22:59:49.745 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 22:59:49.745 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 22:59:49.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 22:59:49.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 22:59:49.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:00:09.748 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:00:09.785 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:00:09.785 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:00:09.785 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:00:09.785 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:00:09.785 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:00:09.785 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:00:09.785 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:00:09.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:00:09.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:00:09.788 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:00:09.788 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:00:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:00:09.788 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:00:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:00:09.788 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:00:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:00:09.788 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:00:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:00:09.788 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:00:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:00:09.788 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:00:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:00:09.788 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:00:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:00:09.788 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:00:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:00:09.788 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:00:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:00:09.788 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:00:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:00:09.788 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:00:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:00:09.788 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:00:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:00:09.788 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:00:09.788 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:00:09.788 139328 ERROR oslo_service.periodic_task >2018-06-24 23:00:39.660 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:00:39.661 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:00:39.670 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:00:39.671 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:00:39.706 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:00:39.706 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:00:39.706 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:00:39.706 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:00:39.706 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:00:39.706 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:00:39.706 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:00:39.708 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:00:39.708 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:00:39.709 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:00:49.710 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:00:49.747 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:00:49.747 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:00:49.747 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:00:49.747 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:00:49.747 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:00:49.747 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:00:49.747 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:00:49.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:00:49.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:00:49.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:01:09.750 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:01:09.786 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:01:09.786 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:01:09.786 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:01:09.786 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:01:09.786 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:01:09.786 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:01:09.786 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:01:09.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:01:09.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:01:09.788 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:01:09.788 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:01:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:01:09.788 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:01:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:01:09.788 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:01:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:01:09.788 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:01:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:01:09.788 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:01:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:01:09.788 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:01:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:01:09.788 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:01:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:01:09.788 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:01:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:01:09.788 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:01:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:01:09.788 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:01:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:01:09.788 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:01:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:01:09.788 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:01:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:01:09.788 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:01:09.788 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:01:09.788 139328 ERROR oslo_service.periodic_task >2018-06-24 23:01:39.660 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:01:39.661 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:01:39.669 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:01:39.670 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:01:39.704 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:01:39.704 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:01:39.704 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:01:39.704 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:01:39.704 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:01:39.704 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:01:39.704 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:01:39.706 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:01:39.706 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:01:39.707 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:01:49.707 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:01:49.743 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:01:49.743 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:01:49.743 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:01:49.743 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:01:49.743 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:01:49.743 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:01:49.743 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:01:49.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:01:49.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:01:49.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:02:09.747 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:02:09.778 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:02:09.778 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:02:09.778 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:02:09.778 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:02:09.778 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:02:09.778 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:02:09.778 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:02:09.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:02:09.781 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:02:09.781 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:02:09.781 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:02:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:02:09.781 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:02:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:02:09.781 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:02:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:02:09.781 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:02:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:02:09.781 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:02:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:02:09.781 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:02:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:02:09.781 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:02:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:02:09.781 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:02:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:02:09.781 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:02:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:02:09.781 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:02:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:02:09.781 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:02:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:02:09.781 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:02:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:02:09.781 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:02:09.781 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:02:09.781 139328 ERROR oslo_service.periodic_task >2018-06-24 23:02:39.663 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:02:39.664 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:02:39.671 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:02:39.672 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:02:39.707 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:02:39.707 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:02:39.707 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:02:39.707 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:02:39.707 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:02:39.707 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:02:39.707 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:02:39.709 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:02:39.709 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:02:39.710 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:02:49.710 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:02:49.748 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:02:49.748 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:02:49.748 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:02:49.748 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:02:49.748 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:02:49.748 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:02:49.748 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:02:49.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:02:49.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:02:49.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:03:09.753 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:03:09.786 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:03:09.786 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:03:09.786 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:03:09.786 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:03:09.786 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:03:09.786 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:03:09.786 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:03:09.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:03:09.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:03:09.789 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:03:09.789 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:03:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:03:09.789 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:03:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:03:09.789 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:03:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:03:09.789 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:03:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:03:09.789 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:03:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:03:09.789 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:03:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:03:09.789 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:03:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:03:09.789 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:03:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:03:09.789 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:03:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:03:09.789 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:03:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:03:09.789 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:03:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:03:09.789 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:03:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:03:09.789 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:03:09.789 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:03:09.789 139328 ERROR oslo_service.periodic_task >2018-06-24 23:03:39.661 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:03:39.662 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:03:39.673 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:03:39.673 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:03:39.708 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:03:39.708 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:03:39.708 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:03:39.708 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:03:39.708 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:03:39.708 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:03:39.708 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:03:39.710 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:03:39.710 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:03:39.711 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:03:49.711 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:03:49.746 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:03:49.746 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:03:49.746 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:03:49.746 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:03:49.746 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:03:49.746 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:03:49.746 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:03:49.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:03:49.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:03:49.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:04:09.750 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:04:09.783 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:04:09.783 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:04:09.783 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:04:09.783 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:04:09.783 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:04:09.783 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:04:09.783 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:04:09.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:04:09.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:04:09.785 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:04:09.785 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:04:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:04:09.785 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:04:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:04:09.785 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:04:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:04:09.785 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:04:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:04:09.785 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:04:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:04:09.785 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:04:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:04:09.785 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:04:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:04:09.785 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:04:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:04:09.785 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:04:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:04:09.785 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:04:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:04:09.785 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:04:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:04:09.785 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:04:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:04:09.785 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:04:09.785 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:04:09.785 139328 ERROR oslo_service.periodic_task >2018-06-24 23:04:39.657 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:04:39.658 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:04:39.666 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:04:39.667 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:04:39.702 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:04:39.702 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:04:39.702 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:04:39.702 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:04:39.702 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:04:39.702 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:04:39.702 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:04:39.704 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:04:39.704 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:04:39.705 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:04:49.705 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:04:49.742 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:04:49.742 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:04:49.742 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:04:49.742 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:04:49.742 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:04:49.742 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:04:49.742 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:04:49.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:04:49.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:04:49.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:05:09.746 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:05:09.782 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:05:09.782 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:05:09.782 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:05:09.782 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:05:09.782 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:05:09.782 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:05:09.782 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:05:09.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:05:09.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:05:09.784 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:05:09.784 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:05:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:05:09.784 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:05:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:05:09.784 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:05:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:05:09.784 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:05:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:05:09.784 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:05:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:05:09.784 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:05:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:05:09.784 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:05:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:05:09.784 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:05:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:05:09.784 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:05:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:05:09.784 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:05:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:05:09.784 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:05:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:05:09.784 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:05:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:05:09.784 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:05:09.784 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:05:09.784 139328 ERROR oslo_service.periodic_task >2018-06-24 23:05:39.657 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:05:39.658 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:05:39.664 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:05:39.665 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:05:39.698 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:05:39.698 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:05:39.698 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:05:39.698 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:05:39.698 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:05:39.698 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:05:39.698 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:05:39.701 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:05:39.701 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:05:39.701 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:05:49.702 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:05:49.735 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:05:49.735 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:05:49.735 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:05:49.735 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:05:49.735 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:05:49.735 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:05:49.735 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:05:49.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:05:49.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:05:49.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:06:09.738 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:06:09.792 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:06:09.792 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:06:09.792 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:06:09.792 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:06:09.792 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:06:09.792 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:06:09.792 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:06:09.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:06:09.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.13 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:06:09.795 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:06:09.795 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:06:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:06:09.795 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:06:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:06:09.795 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:06:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:06:09.795 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:06:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:06:09.795 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:06:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:06:09.795 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:06:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:06:09.795 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:06:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:06:09.795 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:06:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:06:09.795 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:06:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:06:09.795 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:06:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:06:09.795 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:06:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:06:09.795 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:06:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:06:09.795 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:06:09.795 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:06:09.795 139328 ERROR oslo_service.periodic_task >2018-06-24 23:06:39.656 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:06:39.657 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:06:39.664 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:06:39.665 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:06:39.699 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:06:39.699 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:06:39.699 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:06:39.699 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:06:39.699 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:06:39.699 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:06:39.699 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:06:39.701 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:06:39.701 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:06:39.702 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:06:49.702 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:06:49.736 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:06:49.736 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:06:49.736 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:06:49.736 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:06:49.736 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:06:49.736 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:06:49.736 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:06:49.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:06:49.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:06:49.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:07:09.740 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:07:09.776 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:07:09.776 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:07:09.776 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:07:09.776 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:07:09.776 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:07:09.776 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:07:09.776 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:07:09.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:07:09.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:07:09.779 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:07:09.779 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:07:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:07:09.779 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:07:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:07:09.779 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:07:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:07:09.779 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:07:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:07:09.779 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:07:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:07:09.779 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:07:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:07:09.779 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:07:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:07:09.779 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:07:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:07:09.779 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:07:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:07:09.779 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:07:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:07:09.779 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:07:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:07:09.779 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:07:09.779 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:07:09.779 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:07:09.779 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:07:09.779 139328 ERROR oslo_service.periodic_task >2018-06-24 23:07:39.661 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:07:39.662 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:07:39.669 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:07:39.669 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:07:39.705 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:07:39.705 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:07:39.705 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:07:39.705 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:07:39.705 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:07:39.705 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:07:39.705 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:07:39.707 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:07:39.707 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:07:39.708 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:07:49.708 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:07:49.743 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:07:49.743 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:07:49.743 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:07:49.743 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:07:49.743 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:07:49.743 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:07:49.743 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:07:49.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:07:49.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:07:49.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:08:09.747 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:08:09.788 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:08:09.788 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:08:09.788 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:08:09.788 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:08:09.788 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:08:09.788 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:08:09.788 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:08:09.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:08:09.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.12 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:08:09.791 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:08:09.791 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:08:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:08:09.791 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:08:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:08:09.791 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:08:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:08:09.791 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:08:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:08:09.791 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:08:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:08:09.791 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:08:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:08:09.791 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:08:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:08:09.791 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:08:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:08:09.791 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:08:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:08:09.791 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:08:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:08:09.791 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:08:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:08:09.791 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:08:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:08:09.791 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:08:09.791 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:08:09.791 139328 ERROR oslo_service.periodic_task >2018-06-24 23:08:39.662 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:08:39.663 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:08:39.670 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:08:39.670 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:08:39.704 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:08:39.704 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:08:39.704 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:08:39.704 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:08:39.704 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:08:39.704 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:08:39.704 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:08:39.707 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:08:39.707 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:08:39.707 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:08:49.708 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:08:49.741 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:08:49.741 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:08:49.741 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:08:49.741 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:08:49.741 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:08:49.741 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:08:49.741 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:08:49.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:08:49.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:08:49.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:09:09.745 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:09:09.778 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:09:09.778 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:09:09.778 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:09:09.778 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:09:09.778 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:09:09.778 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:09:09.778 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:09:09.781 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:09:09.781 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:09:09.781 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:09:09.781 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:09:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:09:09.781 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:09:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:09:09.781 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:09:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:09:09.781 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:09:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:09:09.781 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:09:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:09:09.781 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:09:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:09:09.781 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:09:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:09:09.781 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:09:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:09:09.781 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:09:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:09:09.781 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:09:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:09:09.781 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:09:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:09:09.781 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:09:09.781 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:09:09.781 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:09:09.781 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:09:09.781 139328 ERROR oslo_service.periodic_task >2018-06-24 23:09:39.663 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:09:39.663 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:09:39.673 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:09:39.674 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:09:39.708 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:09:39.708 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:09:39.708 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:09:39.708 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:09:39.708 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:09:39.708 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:09:39.708 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:09:39.710 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:09:39.711 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:09:39.711 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:09:49.712 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:09:49.752 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:09:49.752 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:09:49.752 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:09:49.752 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:09:49.752 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:09:49.752 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:09:49.752 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:09:49.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:09:49.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.081 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:09:49.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:10:09.756 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:10:09.789 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:10:09.789 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:10:09.789 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:10:09.789 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:10:09.789 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:10:09.789 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:10:09.789 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:10:09.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:10:09.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:10:09.791 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:10:09.791 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:10:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:10:09.791 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:10:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:10:09.791 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:10:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:10:09.791 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:10:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:10:09.791 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:10:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:10:09.791 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:10:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:10:09.791 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:10:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:10:09.791 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:10:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:10:09.791 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:10:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:10:09.791 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:10:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:10:09.791 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:10:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:10:09.791 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:10:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:10:09.791 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:10:09.791 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:10:09.791 139328 ERROR oslo_service.periodic_task >2018-06-24 23:10:39.663 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:10:39.664 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:10:39.675 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:10:39.676 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:10:39.708 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:10:39.708 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:10:39.708 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:10:39.708 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:10:39.708 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:10:39.708 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:10:39.708 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:10:39.711 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:10:39.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:10:39.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:10:49.713 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:10:49.761 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:10:49.761 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:10:49.761 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:10:49.761 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:10:49.761 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:10:49.761 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:10:49.761 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:10:49.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:10:49.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.088 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:10:49.764 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:11:09.764 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:11:09.799 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:11:09.799 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:11:09.799 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:11:09.799 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:11:09.799 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:11:09.799 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:11:09.799 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:11:09.801 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:11:09.801 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.125 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:11:09.801 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:11:09.801 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:11:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:11:09.801 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:11:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:11:09.801 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:11:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:11:09.801 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:11:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:11:09.801 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:11:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:11:09.801 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:11:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:11:09.801 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:11:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:11:09.801 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:11:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:11:09.801 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:11:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:11:09.801 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:11:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:11:09.801 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:11:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:11:09.801 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:11:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:11:09.801 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:11:09.801 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:11:09.801 139328 ERROR oslo_service.periodic_task >2018-06-24 23:11:39.663 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:11:39.664 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:11:39.671 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:11:39.672 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:11:39.704 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:11:39.704 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:11:39.704 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:11:39.704 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:11:39.704 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:11:39.704 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:11:39.704 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:11:39.707 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:11:39.707 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:11:39.707 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:11:49.708 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:11:49.743 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:11:49.743 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:11:49.743 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:11:49.743 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:11:49.743 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:11:49.743 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:11:49.743 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:11:49.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:11:49.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:11:49.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:12:09.747 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:12:09.779 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:12:09.779 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:12:09.779 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:12:09.779 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:12:09.779 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:12:09.779 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:12:09.779 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:12:09.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:12:09.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:12:09.782 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:12:09.782 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:12:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:12:09.782 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:12:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:12:09.782 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:12:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:12:09.782 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:12:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:12:09.782 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:12:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:12:09.782 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:12:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:12:09.782 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:12:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:12:09.782 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:12:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:12:09.782 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:12:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:12:09.782 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:12:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:12:09.782 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:12:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:12:09.782 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:12:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:12:09.782 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:12:09.782 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:12:09.782 139328 ERROR oslo_service.periodic_task >2018-06-24 23:12:39.664 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:12:39.665 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:12:39.672 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:12:39.672 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:12:39.706 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:12:39.706 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:12:39.706 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:12:39.706 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:12:39.706 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:12:39.706 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:12:39.706 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:12:39.708 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:12:39.708 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:12:39.708 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:12:49.709 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:12:49.742 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:12:49.742 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:12:49.742 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:12:49.742 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:12:49.742 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:12:49.742 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:12:49.742 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:12:49.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:12:49.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:12:49.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:13:09.746 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:13:09.782 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:13:09.782 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:13:09.782 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:13:09.782 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:13:09.782 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:13:09.782 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:13:09.782 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:13:09.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:13:09.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:13:09.785 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:13:09.785 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:13:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:13:09.785 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:13:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:13:09.785 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:13:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:13:09.785 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:13:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:13:09.785 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:13:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:13:09.785 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:13:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:13:09.785 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:13:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:13:09.785 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:13:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:13:09.785 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:13:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:13:09.785 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:13:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:13:09.785 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:13:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:13:09.785 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:13:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:13:09.785 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:13:09.785 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:13:09.785 139328 ERROR oslo_service.periodic_task >2018-06-24 23:13:39.666 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:13:39.667 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:13:39.676 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:13:39.677 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:13:39.713 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:13:39.713 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:13:39.713 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:13:39.713 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:13:39.713 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:13:39.713 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:13:39.713 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:13:39.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:13:39.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:13:39.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:13:49.717 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:13:49.751 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:13:49.751 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:13:49.751 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:13:49.751 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:13:49.751 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:13:49.751 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:13:49.751 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:13:49.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:13:49.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:13:49.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:14:09.755 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:14:09.790 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:14:09.790 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:14:09.790 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:14:09.790 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:14:09.790 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:14:09.790 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:14:09.790 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:14:09.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:14:09.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:14:09.793 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:14:09.793 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:14:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:14:09.793 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:14:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:14:09.793 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:14:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:14:09.793 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:14:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:14:09.793 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:14:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:14:09.793 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:14:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:14:09.793 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:14:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:14:09.793 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:14:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:14:09.793 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:14:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:14:09.793 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:14:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:14:09.793 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:14:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:14:09.793 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:14:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:14:09.793 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:14:09.793 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:14:09.793 139328 ERROR oslo_service.periodic_task >2018-06-24 23:14:39.664 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:14:39.665 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:14:39.672 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:14:39.673 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:14:39.706 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:14:39.706 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:14:39.706 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:14:39.706 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:14:39.706 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:14:39.706 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:14:39.706 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:14:39.709 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:14:39.709 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:14:39.710 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:14:49.710 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:14:49.745 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:14:49.745 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:14:49.745 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:14:49.745 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:14:49.745 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:14:49.745 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:14:49.745 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:14:49.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:14:49.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:14:49.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:15:09.749 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:15:09.782 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:15:09.782 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:15:09.782 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:15:09.782 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:15:09.782 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:15:09.782 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:15:09.782 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:15:09.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:15:09.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:15:09.785 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:15:09.785 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:15:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:15:09.785 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:15:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:15:09.785 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:15:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:15:09.785 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:15:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:15:09.785 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:15:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:15:09.785 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:15:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:15:09.785 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:15:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:15:09.785 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:15:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:15:09.785 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:15:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:15:09.785 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:15:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:15:09.785 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:15:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:15:09.785 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:15:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:15:09.785 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:15:09.785 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:15:09.785 139328 ERROR oslo_service.periodic_task >2018-06-24 23:15:39.666 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:15:39.667 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:15:39.673 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:15:39.674 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:15:39.709 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:15:39.709 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:15:39.709 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:15:39.709 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:15:39.709 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:15:39.709 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:15:39.709 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:15:39.711 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:15:39.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:15:39.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:15:49.713 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:15:49.746 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:15:49.746 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:15:49.746 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:15:49.746 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:15:49.746 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:15:49.746 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:15:49.746 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:15:49.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:15:49.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:15:49.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:16:09.749 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:16:09.782 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:16:09.782 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:16:09.782 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:16:09.782 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:16:09.782 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:16:09.782 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:16:09.782 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:16:09.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:16:09.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:16:09.785 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:16:09.785 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:16:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:16:09.785 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:16:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:16:09.785 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:16:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:16:09.785 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:16:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:16:09.785 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:16:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:16:09.785 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:16:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:16:09.785 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:16:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:16:09.785 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:16:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:16:09.785 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:16:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:16:09.785 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:16:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:16:09.785 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:16:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:16:09.785 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:16:09.785 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:16:09.785 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:16:09.785 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:16:09.785 139328 ERROR oslo_service.periodic_task >2018-06-24 23:16:39.666 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:16:39.667 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:16:39.675 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:16:39.676 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:16:39.713 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:16:39.713 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:16:39.713 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:16:39.713 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:16:39.713 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:16:39.713 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:16:39.713 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:16:39.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:16:39.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:16:39.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:16:49.717 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:16:49.753 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:16:49.753 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:16:49.753 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:16:49.753 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:16:49.753 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:16:49.753 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:16:49.753 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:16:49.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:16:49.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.08 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:16:49.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:17:09.756 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:17:09.790 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:17:09.790 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:17:09.790 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:17:09.790 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:17:09.790 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:17:09.790 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:17:09.790 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:17:09.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:17:09.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:17:09.793 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:17:09.793 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:17:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:17:09.793 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:17:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:17:09.793 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:17:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:17:09.793 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:17:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:17:09.793 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:17:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:17:09.793 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:17:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:17:09.793 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:17:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:17:09.793 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:17:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:17:09.793 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:17:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:17:09.793 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:17:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:17:09.793 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:17:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:17:09.793 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:17:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:17:09.793 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:17:09.793 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:17:09.793 139328 ERROR oslo_service.periodic_task >2018-06-24 23:17:39.665 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:17:39.666 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:17:39.674 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:17:39.674 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:17:39.709 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:17:39.709 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:17:39.709 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:17:39.709 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:17:39.709 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:17:39.709 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:17:39.709 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:17:39.711 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:17:39.711 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:17:39.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:17:49.712 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:17:49.747 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:17:49.747 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:17:49.747 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:17:49.747 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:17:49.747 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:17:49.747 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:17:49.747 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:17:49.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:17:49.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:17:49.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:18:09.750 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:18:09.784 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:18:09.784 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:18:09.784 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:18:09.784 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:18:09.784 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:18:09.784 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:18:09.784 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:18:09.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:18:09.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:18:09.787 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:18:09.787 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:18:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:18:09.787 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:18:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:18:09.787 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:18:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:18:09.787 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:18:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:18:09.787 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:18:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:18:09.787 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:18:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:18:09.787 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:18:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:18:09.787 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:18:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:18:09.787 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:18:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:18:09.787 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:18:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:18:09.787 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:18:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:18:09.787 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:18:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:18:09.787 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:18:09.787 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:18:09.787 139328 ERROR oslo_service.periodic_task >2018-06-24 23:18:39.668 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:18:39.669 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:18:39.676 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:18:39.676 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:18:39.710 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:18:39.710 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:18:39.710 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:18:39.710 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:18:39.710 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:18:39.710 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:18:39.710 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:18:39.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:18:39.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:18:39.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:18:49.714 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:18:49.750 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:18:49.750 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:18:49.750 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:18:49.750 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:18:49.750 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:18:49.750 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:18:49.750 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:18:49.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:18:49.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:18:49.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:19:09.755 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:19:09.787 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:19:09.787 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:19:09.787 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:19:09.787 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:19:09.787 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:19:09.787 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:19:09.787 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:19:09.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:19:09.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:19:09.790 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:19:09.790 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:19:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:19:09.790 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:19:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:19:09.790 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:19:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:19:09.790 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:19:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:19:09.790 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:19:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:19:09.790 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:19:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:19:09.790 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:19:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:19:09.790 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:19:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:19:09.790 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:19:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:19:09.790 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:19:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:19:09.790 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:19:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:19:09.790 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:19:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:19:09.790 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:19:09.790 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:19:09.790 139328 ERROR oslo_service.periodic_task >2018-06-24 23:19:39.671 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:19:39.672 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:19:39.681 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:19:39.681 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:19:39.721 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:19:39.721 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:19:39.721 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:19:39.721 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:19:39.721 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:19:39.721 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:19:39.721 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:19:39.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:19:39.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.042 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:19:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:19:49.724 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:19:49.757 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:19:49.757 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:19:49.757 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:19:49.757 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:19:49.757 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:19:49.757 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:19:49.757 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:19:49.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:19:49.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:19:49.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:20:09.761 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:20:09.795 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:20:09.795 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:20:09.795 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:20:09.795 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:20:09.795 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:20:09.795 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:20:09.795 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:20:09.797 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:20:09.797 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:20:09.797 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:20:09.797 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:20:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:20:09.797 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:20:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:20:09.797 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:20:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:20:09.797 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:20:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:20:09.797 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:20:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:20:09.797 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:20:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:20:09.797 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:20:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:20:09.797 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:20:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:20:09.797 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:20:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:20:09.797 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:20:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:20:09.797 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:20:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:20:09.797 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:20:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:20:09.797 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:20:09.797 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:20:09.797 139328 ERROR oslo_service.periodic_task >2018-06-24 23:20:39.669 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:20:39.670 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:20:39.676 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:20:39.677 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:20:39.715 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:20:39.715 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:20:39.715 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:20:39.715 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:20:39.715 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:20:39.715 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:20:39.715 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:20:39.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:20:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.041 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:20:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:20:49.719 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:20:49.754 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:20:49.754 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:20:49.754 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:20:49.754 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:20:49.754 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:20:49.754 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:20:49.754 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:20:49.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:20:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.08 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:20:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:21:09.758 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:21:09.792 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:21:09.792 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:21:09.792 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:21:09.792 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:21:09.792 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:21:09.792 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:21:09.792 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:21:09.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:21:09.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.118 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:21:09.796 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:21:09.796 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:21:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:21:09.796 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:21:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:21:09.796 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:21:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:21:09.796 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:21:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:21:09.796 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:21:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:21:09.796 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:21:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:21:09.796 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:21:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:21:09.796 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:21:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:21:09.796 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:21:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:21:09.796 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:21:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:21:09.796 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:21:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:21:09.796 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:21:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:21:09.796 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:21:09.796 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:21:09.796 139328 ERROR oslo_service.periodic_task >2018-06-24 23:21:39.668 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:21:39.669 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:21:39.680 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:21:39.680 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:21:39.715 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:21:39.715 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:21:39.715 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:21:39.715 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:21:39.715 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:21:39.715 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:21:39.715 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:21:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:21:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:21:39.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:21:49.719 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:21:49.757 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:21:49.757 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:21:49.757 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:21:49.757 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:21:49.757 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:21:49.757 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:21:49.757 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:21:49.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:21:49.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:21:49.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:22:09.760 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:22:09.794 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:22:09.794 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:22:09.794 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:22:09.794 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:22:09.794 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:22:09.794 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:22:09.794 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:22:09.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:22:09.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:22:09.796 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:22:09.796 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:22:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:22:09.796 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:22:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:22:09.796 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:22:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:22:09.796 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:22:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:22:09.796 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:22:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:22:09.796 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:22:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:22:09.796 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:22:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:22:09.796 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:22:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:22:09.796 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:22:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:22:09.796 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:22:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:22:09.796 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:22:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:22:09.796 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:22:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:22:09.796 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:22:09.796 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:22:09.796 139328 ERROR oslo_service.periodic_task >2018-06-24 23:22:39.668 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:22:39.669 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:22:39.675 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:22:39.676 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:22:39.709 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:22:39.709 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:22:39.709 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:22:39.709 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:22:39.709 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:22:39.709 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:22:39.709 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:22:39.711 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:22:39.711 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:22:39.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:22:49.712 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:22:49.747 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:22:49.747 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:22:49.747 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:22:49.747 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:22:49.747 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:22:49.747 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:22:49.747 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:22:49.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:22:49.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:22:49.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:23:09.751 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:23:09.791 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:23:09.791 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:23:09.791 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:23:09.791 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:23:09.791 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:23:09.791 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:23:09.791 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:23:09.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:23:09.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.119 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:23:09.796 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:23:09.796 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:23:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:23:09.796 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:23:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:23:09.796 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:23:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:23:09.796 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:23:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:23:09.796 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:23:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:23:09.796 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:23:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:23:09.796 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:23:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:23:09.796 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:23:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:23:09.796 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:23:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:23:09.796 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:23:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:23:09.796 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:23:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:23:09.796 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:23:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:23:09.796 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:23:09.796 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:23:09.796 139328 ERROR oslo_service.periodic_task >2018-06-24 23:23:39.668 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:23:39.669 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:23:39.677 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:23:39.677 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:23:39.712 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:23:39.712 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:23:39.712 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:23:39.712 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:23:39.712 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:23:39.712 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:23:39.712 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:23:39.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:23:39.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:23:39.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:23:49.716 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:23:49.748 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:23:49.748 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:23:49.748 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:23:49.748 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:23:49.748 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:23:49.748 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:23:49.748 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:23:49.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:23:49.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:23:49.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:24:09.752 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:24:09.792 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:24:09.792 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:24:09.792 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:24:09.792 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:24:09.792 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:24:09.792 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:24:09.792 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:24:09.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:24:09.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:24:09.796 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:24:09.796 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:24:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:24:09.796 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:24:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:24:09.796 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:24:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:24:09.796 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:24:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:24:09.796 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:24:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:24:09.796 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:24:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:24:09.796 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:24:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:24:09.796 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:24:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:24:09.796 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:24:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:24:09.796 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:24:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:24:09.796 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:24:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:24:09.796 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:24:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:24:09.796 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:24:09.796 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:24:09.796 139328 ERROR oslo_service.periodic_task >2018-06-24 23:24:39.667 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:24:39.668 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:24:39.677 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:24:39.677 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:24:39.715 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:24:39.715 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:24:39.715 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:24:39.715 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:24:39.715 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:24:39.715 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:24:39.715 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:24:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:24:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:24:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:24:49.719 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:24:49.755 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:24:49.755 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:24:49.755 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:24:49.755 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:24:49.755 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:24:49.755 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:24:49.755 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:24:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:24:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:24:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:25:09.758 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:25:09.794 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:25:09.794 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:25:09.794 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:25:09.794 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:25:09.794 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:25:09.794 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:25:09.794 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:25:09.797 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:25:09.797 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.119 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:25:09.797 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:25:09.797 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:25:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:25:09.797 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:25:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:25:09.797 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:25:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:25:09.797 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:25:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:25:09.797 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:25:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:25:09.797 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:25:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:25:09.797 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:25:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:25:09.797 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:25:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:25:09.797 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:25:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:25:09.797 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:25:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:25:09.797 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:25:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:25:09.797 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:25:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:25:09.797 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:25:09.797 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:25:09.797 139328 ERROR oslo_service.periodic_task >2018-06-24 23:25:39.669 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:25:39.670 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:25:39.678 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:25:39.678 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:25:39.711 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:25:39.711 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:25:39.711 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:25:39.711 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:25:39.711 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:25:39.711 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:25:39.711 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:25:39.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:25:39.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:25:39.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:25:49.714 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:25:49.746 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:25:49.746 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:25:49.746 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:25:49.746 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:25:49.746 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:25:49.746 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:25:49.746 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:25:49.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:25:49.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.07 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:25:49.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:26:09.749 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:26:09.782 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:26:09.782 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:26:09.782 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:26:09.782 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:26:09.782 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:26:09.782 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:26:09.782 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:26:09.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:26:09.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.105 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:26:09.784 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:26:09.784 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:26:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:26:09.784 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:26:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:26:09.784 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:26:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:26:09.784 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:26:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:26:09.784 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:26:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:26:09.784 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:26:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:26:09.784 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:26:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:26:09.784 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:26:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:26:09.784 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:26:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:26:09.784 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:26:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:26:09.784 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:26:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:26:09.784 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:26:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:26:09.784 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:26:09.784 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:26:09.784 139328 ERROR oslo_service.periodic_task >2018-06-24 23:26:39.666 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:26:39.667 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:26:39.674 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:26:39.675 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:26:39.711 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:26:39.711 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:26:39.711 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:26:39.711 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:26:39.711 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:26:39.711 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:26:39.711 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:26:39.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:26:39.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:26:39.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:26:49.714 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:26:49.748 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:26:49.748 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:26:49.748 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:26:49.748 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:26:49.748 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:26:49.748 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:26:49.748 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:26:49.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:26:49.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:26:49.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:27:09.752 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:27:09.789 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:27:09.789 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:27:09.789 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:27:09.789 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:27:09.789 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:27:09.789 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:27:09.789 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:27:09.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:27:09.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:27:09.791 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:27:09.791 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:27:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:27:09.791 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:27:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:27:09.791 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:27:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:27:09.791 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:27:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:27:09.791 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:27:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:27:09.791 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:27:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:27:09.791 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:27:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:27:09.791 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:27:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:27:09.791 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:27:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:27:09.791 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:27:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:27:09.791 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:27:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:27:09.791 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:27:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:27:09.791 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:27:09.791 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:27:09.791 139328 ERROR oslo_service.periodic_task >2018-06-24 23:27:39.663 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:27:39.663 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:27:39.671 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:27:39.671 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:27:39.706 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:27:39.706 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:27:39.706 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:27:39.706 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:27:39.706 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:27:39.706 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:27:39.706 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:27:39.708 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:27:39.708 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:27:39.708 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:27:49.709 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:27:49.743 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:27:49.743 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:27:49.743 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:27:49.743 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:27:49.743 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:27:49.743 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:27:49.743 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:27:49.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:27:49.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:27:49.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:28:09.745 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:28:09.779 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:28:09.779 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:28:09.779 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:28:09.779 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:28:09.779 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:28:09.779 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:28:09.779 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:28:09.781 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:28:09.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:28:09.782 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:28:09.782 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:28:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:28:09.782 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:28:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:28:09.782 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:28:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:28:09.782 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:28:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:28:09.782 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:28:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:28:09.782 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:28:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:28:09.782 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:28:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:28:09.782 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:28:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:28:09.782 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:28:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:28:09.782 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:28:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:28:09.782 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:28:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:28:09.782 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:28:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:28:09.782 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:28:09.782 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:28:09.782 139328 ERROR oslo_service.periodic_task >2018-06-24 23:28:39.664 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:28:39.665 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:28:39.672 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:28:39.673 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:28:39.709 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:28:39.709 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:28:39.709 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:28:39.709 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:28:39.709 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:28:39.709 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:28:39.709 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:28:39.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:28:39.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:28:39.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:28:49.714 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:28:49.746 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:28:49.746 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:28:49.746 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:28:49.746 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:28:49.746 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:28:49.746 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:28:49.746 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:28:49.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:28:49.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:28:49.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:29:09.750 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:29:09.802 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:29:09.802 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:29:09.802 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:29:09.802 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:29:09.802 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:29:09.802 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:29:09.802 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:29:09.804 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:29:09.804 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.131 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:29:09.804 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:29:09.804 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:29:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:29:09.804 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:29:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:29:09.804 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:29:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:29:09.804 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:29:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:29:09.804 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:29:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:29:09.804 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:29:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:29:09.804 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:29:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:29:09.804 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:29:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:29:09.804 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:29:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:29:09.804 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:29:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:29:09.804 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:29:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:29:09.804 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:29:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:29:09.804 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:29:09.804 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:29:09.804 139328 ERROR oslo_service.periodic_task >2018-06-24 23:29:39.666 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:29:39.667 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:29:39.673 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:29:39.673 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:29:39.711 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:29:39.711 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:29:39.711 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:29:39.711 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:29:39.711 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:29:39.711 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:29:39.711 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:29:39.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:29:39.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:29:39.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:29:49.715 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:29:49.750 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:29:49.750 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:29:49.750 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:29:49.750 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:29:49.750 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:29:49.750 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:29:49.750 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:29:49.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:29:49.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:29:49.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:30:09.753 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:30:09.791 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:30:09.791 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:30:09.791 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:30:09.791 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:30:09.791 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:30:09.791 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:30:09.791 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:30:09.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:30:09.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.12 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:30:09.794 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:30:09.794 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:30:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:30:09.794 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:30:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:30:09.794 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:30:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:30:09.794 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:30:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:30:09.794 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:30:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:30:09.794 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:30:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:30:09.794 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:30:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:30:09.794 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:30:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:30:09.794 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:30:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:30:09.794 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:30:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:30:09.794 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:30:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:30:09.794 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:30:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:30:09.794 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:30:09.794 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:30:09.794 139328 ERROR oslo_service.periodic_task >2018-06-24 23:30:39.667 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:30:39.668 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:30:39.677 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:30:39.678 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:30:39.713 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:30:39.713 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:30:39.713 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:30:39.713 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:30:39.713 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:30:39.713 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:30:39.713 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:30:39.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:30:39.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:30:39.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:30:49.718 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:30:49.752 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:30:49.752 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:30:49.752 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:30:49.752 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:30:49.752 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:30:49.752 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:30:49.752 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:30:49.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:30:49.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:30:49.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:31:09.755 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:31:09.788 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:31:09.788 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:31:09.788 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:31:09.788 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:31:09.788 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:31:09.788 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:31:09.788 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:31:09.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:31:09.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:31:09.790 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:31:09.790 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:31:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:31:09.790 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:31:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:31:09.790 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:31:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:31:09.790 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:31:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:31:09.790 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:31:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:31:09.790 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:31:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:31:09.790 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:31:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:31:09.790 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:31:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:31:09.790 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:31:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:31:09.790 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:31:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:31:09.790 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:31:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:31:09.790 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:31:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:31:09.790 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:31:09.790 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:31:09.790 139328 ERROR oslo_service.periodic_task >2018-06-24 23:31:39.662 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:31:39.663 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:31:39.672 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:31:39.673 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:31:39.709 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:31:39.709 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:31:39.709 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:31:39.709 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:31:39.709 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:31:39.709 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:31:39.709 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:31:39.711 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:31:39.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:31:39.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:31:49.712 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:31:49.749 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:31:49.749 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:31:49.749 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:31:49.749 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:31:49.749 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:31:49.749 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:31:49.749 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:31:49.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:31:49.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:31:49.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:32:09.753 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:32:09.787 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:32:09.787 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:32:09.787 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:32:09.787 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:32:09.787 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:32:09.787 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:32:09.787 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:32:09.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:32:09.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:32:09.790 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:32:09.790 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:32:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:32:09.790 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:32:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:32:09.790 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:32:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:32:09.790 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:32:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:32:09.790 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:32:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:32:09.790 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:32:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:32:09.790 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:32:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:32:09.790 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:32:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:32:09.790 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:32:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:32:09.790 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:32:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:32:09.790 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:32:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:32:09.790 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:32:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:32:09.790 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:32:09.790 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:32:09.790 139328 ERROR oslo_service.periodic_task >2018-06-24 23:32:39.662 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:32:39.663 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:32:39.671 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:32:39.672 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:32:39.707 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:32:39.707 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:32:39.707 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:32:39.707 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:32:39.707 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:32:39.707 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:32:39.707 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:32:39.710 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:32:39.710 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:32:39.711 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:32:49.711 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:32:49.744 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:32:49.744 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:32:49.744 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:32:49.744 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:32:49.744 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:32:49.744 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:32:49.744 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:32:49.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:32:49.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:32:49.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:33:09.747 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:33:09.781 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:33:09.781 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:33:09.781 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:33:09.781 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:33:09.781 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:33:09.781 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:33:09.781 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:33:09.783 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:33:09.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:33:09.784 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:33:09.784 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:33:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:33:09.784 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:33:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:33:09.784 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:33:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:33:09.784 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:33:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:33:09.784 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:33:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:33:09.784 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:33:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:33:09.784 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:33:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:33:09.784 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:33:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:33:09.784 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:33:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:33:09.784 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:33:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:33:09.784 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:33:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:33:09.784 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:33:09.784 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:33:09.784 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:33:09.784 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:33:09.784 139328 ERROR oslo_service.periodic_task >2018-06-24 23:33:39.666 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:33:39.666 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:33:39.674 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:33:39.675 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:33:39.710 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:33:39.710 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:33:39.710 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:33:39.710 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:33:39.710 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:33:39.710 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:33:39.710 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:33:39.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:33:39.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:33:39.713 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:33:49.713 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:33:49.747 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:33:49.747 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:33:49.747 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:33:49.747 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:33:49.747 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:33:49.747 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:33:49.747 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:33:49.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:33:49.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:33:49.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:34:09.750 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:34:09.783 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:34:09.783 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:34:09.783 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:34:09.783 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:34:09.783 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:34:09.783 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:34:09.783 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:34:09.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:34:09.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:34:09.786 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:34:09.786 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:34:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:34:09.786 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:34:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:34:09.786 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:34:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:34:09.786 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:34:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:34:09.786 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:34:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:34:09.786 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:34:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:34:09.786 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:34:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:34:09.786 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:34:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:34:09.786 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:34:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:34:09.786 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:34:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:34:09.786 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:34:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:34:09.786 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:34:09.786 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:34:09.786 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:34:09.786 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:34:09.786 139328 ERROR oslo_service.periodic_task >2018-06-24 23:34:39.668 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:34:39.669 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:34:39.676 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:34:39.676 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:34:39.707 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:34:39.707 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:34:39.707 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:34:39.707 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:34:39.707 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:34:39.707 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:34:39.707 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:34:39.709 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:34:39.709 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.032 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:34:39.710 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:34:49.710 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:34:49.742 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:34:49.742 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:34:49.742 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:34:49.742 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:34:49.742 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:34:49.742 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:34:49.742 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:34:49.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:34:49.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.068 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:34:49.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:35:09.746 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:35:09.779 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:35:09.779 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:35:09.779 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:35:09.779 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:35:09.779 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:35:09.779 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:35:09.779 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:35:09.781 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:35:09.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.105 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:35:09.782 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:35:09.782 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:35:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:35:09.782 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:35:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:35:09.782 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:35:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:35:09.782 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:35:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:35:09.782 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:35:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:35:09.782 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:35:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:35:09.782 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:35:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:35:09.782 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:35:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:35:09.782 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:35:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:35:09.782 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:35:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:35:09.782 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:35:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:35:09.782 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:35:09.782 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:35:09.782 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:35:09.782 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:35:09.782 139328 ERROR oslo_service.periodic_task >2018-06-24 23:35:39.664 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:35:39.665 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:35:39.672 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:35:39.673 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:35:39.707 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:35:39.707 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:35:39.707 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:35:39.707 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:35:39.707 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:35:39.707 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:35:39.707 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:35:39.710 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:35:39.710 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:35:39.710 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:35:49.711 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:35:49.745 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:35:49.745 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:35:49.745 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:35:49.745 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:35:49.745 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:35:49.745 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:35:49.745 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:35:49.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:35:49.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:35:49.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:36:09.748 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:36:09.785 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:36:09.785 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:36:09.785 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:36:09.785 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:36:09.785 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:36:09.785 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:36:09.785 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:36:09.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:36:09.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:36:09.787 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:36:09.787 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:36:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:36:09.787 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:36:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:36:09.787 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:36:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:36:09.787 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:36:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:36:09.787 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:36:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:36:09.787 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:36:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:36:09.787 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:36:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:36:09.787 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:36:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:36:09.787 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:36:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:36:09.787 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:36:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:36:09.787 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:36:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:36:09.787 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:36:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:36:09.787 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:36:09.787 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:36:09.787 139328 ERROR oslo_service.periodic_task >2018-06-24 23:36:39.669 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:36:39.670 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:36:39.676 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:36:39.677 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:36:39.711 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:36:39.711 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:36:39.711 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:36:39.711 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:36:39.711 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:36:39.711 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:36:39.711 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:36:39.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:36:39.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:36:39.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:36:49.715 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:36:49.749 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:36:49.749 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:36:49.749 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:36:49.749 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:36:49.749 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:36:49.749 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:36:49.749 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:36:49.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:36:49.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:36:49.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:37:09.752 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:37:09.789 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:37:09.789 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:37:09.789 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:37:09.789 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:37:09.789 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:37:09.789 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:37:09.789 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:37:09.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:37:09.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:37:09.792 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:37:09.792 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:37:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:37:09.792 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:37:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:37:09.792 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:37:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:37:09.792 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:37:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:37:09.792 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:37:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:37:09.792 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:37:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:37:09.792 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:37:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:37:09.792 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:37:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:37:09.792 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:37:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:37:09.792 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:37:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:37:09.792 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:37:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:37:09.792 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:37:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:37:09.792 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:37:09.792 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:37:09.792 139328 ERROR oslo_service.periodic_task >2018-06-24 23:37:39.674 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:37:39.675 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:37:39.681 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:37:39.682 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:37:39.716 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:37:39.716 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:37:39.716 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:37:39.716 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:37:39.716 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:37:39.716 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:37:39.716 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:37:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:37:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:37:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:37:49.719 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:37:49.754 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:37:49.754 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:37:49.754 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:37:49.754 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:37:49.754 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:37:49.754 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:37:49.754 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:37:49.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:37:49.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:37:49.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:38:09.757 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:38:09.794 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:38:09.794 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:38:09.794 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:38:09.794 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:38:09.794 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:38:09.794 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:38:09.794 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:38:09.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:38:09.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:38:09.797 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:38:09.797 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:38:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:38:09.797 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:38:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:38:09.797 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:38:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:38:09.797 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:38:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:38:09.797 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:38:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:38:09.797 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:38:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:38:09.797 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:38:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:38:09.797 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:38:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:38:09.797 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:38:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:38:09.797 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:38:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:38:09.797 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:38:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:38:09.797 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:38:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:38:09.797 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:38:09.797 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:38:09.797 139328 ERROR oslo_service.periodic_task >2018-06-24 23:38:39.678 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:38:39.679 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:38:39.686 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:38:39.686 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:38:39.719 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:38:39.719 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:38:39.719 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:38:39.719 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:38:39.719 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:38:39.719 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:38:39.719 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:38:39.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:38:39.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:38:39.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:38:49.723 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:38:49.755 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:38:49.755 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:38:49.755 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:38:49.755 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:38:49.755 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:38:49.755 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:38:49.755 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:38:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:38:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.07 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:38:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:39:09.758 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:39:09.790 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:39:09.790 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:39:09.790 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:39:09.790 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:39:09.790 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:39:09.790 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:39:09.790 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:39:09.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:39:09.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.105 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:39:09.792 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:39:09.792 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:39:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:39:09.792 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:39:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:39:09.792 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:39:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:39:09.792 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:39:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:39:09.792 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:39:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:39:09.792 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:39:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:39:09.792 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:39:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:39:09.792 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:39:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:39:09.792 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:39:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:39:09.792 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:39:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:39:09.792 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:39:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:39:09.792 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:39:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:39:09.792 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:39:09.792 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:39:09.792 139328 ERROR oslo_service.periodic_task >2018-06-24 23:39:39.674 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:39:39.675 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:39:39.683 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:39:39.683 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:39:39.716 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:39:39.716 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:39:39.716 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:39:39.716 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:39:39.716 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:39:39.716 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:39:39.716 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:39:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:39:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:39:39.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:39:49.719 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:39:49.752 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:39:49.752 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:39:49.752 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:39:49.752 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:39:49.752 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:39:49.752 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:39:49.752 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:39:49.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:39:49.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:39:49.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:40:09.755 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:40:09.788 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:40:09.788 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:40:09.788 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:40:09.788 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:40:09.788 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:40:09.788 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:40:09.788 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:40:09.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:40:09.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.106 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:40:09.790 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:40:09.790 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:40:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:40:09.790 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:40:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:40:09.790 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:40:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:40:09.790 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:40:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:40:09.790 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:40:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:40:09.790 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:40:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:40:09.790 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:40:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:40:09.790 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:40:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:40:09.790 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:40:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:40:09.790 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:40:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:40:09.790 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:40:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:40:09.790 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:40:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:40:09.790 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:40:09.790 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:40:09.790 139328 ERROR oslo_service.periodic_task >2018-06-24 23:40:39.672 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:40:39.672 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:41:39.674 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:41:39.676 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:41:39.682 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:41:39.683 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:41:39.717 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:41:39.717 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:41:39.717 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:41:39.717 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:41:39.717 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:41:39.717 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:41:39.717 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:41:39.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:41:39.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:41:39.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:41:49.721 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:41:49.770 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:41:49.770 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:41:49.770 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:41:49.770 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:41:49.770 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:41:49.770 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:41:49.770 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:41:49.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:41:49.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.089 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:41:49.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:42:09.773 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:42:09.812 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:42:09.812 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:42:09.812 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:42:09.812 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:42:09.812 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:42:09.812 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:42:09.812 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:42:09.814 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:42:09.814 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.131 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:42:09.815 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:42:09.815 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:42:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:42:09.815 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:42:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:42:09.815 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:42:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:42:09.815 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:42:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:42:09.815 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:42:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:42:09.815 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:42:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:42:09.815 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:42:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:42:09.815 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:42:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:42:09.815 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:42:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:42:09.815 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:42:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:42:09.815 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:42:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:42:09.815 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:42:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:42:09.815 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:42:09.815 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:42:09.815 139328 ERROR oslo_service.periodic_task >2018-06-24 23:42:39.676 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:42:39.677 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:42:39.684 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:42:39.685 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:42:39.719 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:42:39.719 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:42:39.719 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:42:39.719 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:42:39.719 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:42:39.719 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:42:39.719 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:42:39.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:42:39.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:42:39.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:42:49.723 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:42:49.757 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:42:49.757 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:42:49.757 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:42:49.757 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:42:49.757 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:42:49.757 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:42:49.757 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:42:49.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:42:49.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:42:49.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:43:09.761 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:43:09.794 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:43:09.794 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:43:09.794 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:43:09.794 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:43:09.794 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:43:09.794 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:43:09.794 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:43:09.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:43:09.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:43:09.796 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:43:09.796 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:43:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:43:09.796 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:43:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:43:09.796 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:43:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:43:09.796 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:43:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:43:09.796 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:43:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:43:09.796 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:43:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:43:09.796 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:43:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:43:09.796 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:43:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:43:09.796 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:43:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:43:09.796 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:43:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:43:09.796 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:43:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:43:09.796 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:43:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:43:09.796 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:43:09.796 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:43:09.796 139328 ERROR oslo_service.periodic_task >2018-06-24 23:43:39.678 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:43:39.679 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:43:39.685 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:43:39.686 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:43:39.719 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:43:39.719 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:43:39.719 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:43:39.719 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:43:39.719 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:43:39.719 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:43:39.719 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:43:39.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:43:39.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:43:39.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:43:49.723 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:43:49.766 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:43:49.766 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:43:49.766 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:43:49.766 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:43:49.766 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:43:49.766 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:43:49.766 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:43:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:43:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.083 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:43:49.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:44:09.770 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:44:09.803 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:44:09.803 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:44:09.803 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:44:09.803 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:44:09.803 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:44:09.803 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:44:09.803 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:44:09.805 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:44:09.805 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.119 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:44:09.806 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:44:09.806 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:44:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:44:09.806 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:44:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:44:09.806 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:44:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:44:09.806 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:44:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:44:09.806 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:44:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:44:09.806 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:44:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:44:09.806 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:44:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:44:09.806 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:44:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:44:09.806 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:44:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:44:09.806 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:44:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:44:09.806 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:44:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:44:09.806 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:44:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:44:09.806 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:44:09.806 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:44:09.806 139328 ERROR oslo_service.periodic_task >2018-06-24 23:44:39.677 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:44:39.678 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:44:39.689 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:44:39.690 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:44:39.730 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:44:39.730 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:44:39.730 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:44:39.730 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:44:39.730 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:44:39.730 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:44:39.730 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:44:39.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:44:39.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.043 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:44:39.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:44:49.734 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:44:49.765 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:44:49.765 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:44:49.765 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:44:49.765 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:44:49.765 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:44:49.765 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:44:49.765 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:44:49.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:44:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:44:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:45:09.768 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:45:09.802 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:45:09.802 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:45:09.802 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:45:09.802 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:45:09.802 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:45:09.802 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:45:09.802 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:45:09.804 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:45:09.804 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:45:09.804 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:45:09.804 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:45:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:45:09.804 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:45:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:45:09.804 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:45:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:45:09.804 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:45:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:45:09.804 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:45:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:45:09.804 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:45:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:45:09.804 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:45:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:45:09.804 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:45:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:45:09.804 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:45:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:45:09.804 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:45:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:45:09.804 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:45:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:45:09.804 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:45:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:45:09.804 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:45:09.804 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:45:09.804 139328 ERROR oslo_service.periodic_task >2018-06-24 23:45:39.676 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:45:39.677 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:45:39.685 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:45:39.686 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:45:39.723 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:45:39.723 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:45:39.723 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:45:39.723 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:45:39.723 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:45:39.723 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:45:39.723 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:45:39.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:45:39.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:45:39.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:45:49.727 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:45:49.761 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:45:49.761 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:45:49.761 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:45:49.761 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:45:49.761 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:45:49.761 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:45:49.761 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:45:49.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:45:49.764 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:45:49.764 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:46:09.764 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:46:09.797 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:46:09.797 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:46:09.797 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:46:09.797 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:46:09.797 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:46:09.797 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:46:09.797 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:46:09.799 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:46:09.799 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:46:09.800 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:46:09.800 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:46:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:46:09.800 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:46:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:46:09.800 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:46:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:46:09.800 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:46:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:46:09.800 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:46:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:46:09.800 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:46:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:46:09.800 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:46:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:46:09.800 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:46:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:46:09.800 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:46:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:46:09.800 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:46:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:46:09.800 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:46:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:46:09.800 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:46:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:46:09.800 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:46:09.800 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:46:09.800 139328 ERROR oslo_service.periodic_task >2018-06-24 23:46:39.671 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:46:39.672 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:46:39.679 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:46:39.679 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:46:39.714 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:46:39.714 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:46:39.714 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:46:39.714 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:46:39.714 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:46:39.714 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:46:39.714 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:46:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:46:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:46:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:46:49.719 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:46:49.754 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:46:49.754 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:46:49.754 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:46:49.754 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:46:49.754 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:46:49.754 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:46:49.754 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:46:49.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:46:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:46:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:47:09.757 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:47:09.790 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:47:09.790 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:47:09.790 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:47:09.790 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:47:09.790 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:47:09.790 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:47:09.790 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:47:09.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:47:09.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:47:09.792 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:47:09.792 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:47:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:47:09.792 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:47:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:47:09.792 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:47:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:47:09.792 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:47:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:47:09.792 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:47:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:47:09.792 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:47:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:47:09.792 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:47:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:47:09.792 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:47:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:47:09.792 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:47:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:47:09.792 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:47:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:47:09.792 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:47:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:47:09.792 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:47:09.792 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:47:09.792 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:47:09.792 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:47:09.792 139328 ERROR oslo_service.periodic_task >2018-06-24 23:47:39.674 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:47:39.675 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:47:39.683 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:47:39.683 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:47:39.716 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:47:39.716 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:47:39.716 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:47:39.716 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:47:39.716 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:47:39.716 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:47:39.716 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:47:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:47:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:47:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:47:49.719 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:47:49.757 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:47:49.757 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:47:49.757 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:47:49.757 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:47:49.757 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:47:49.757 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:47:49.757 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:47:49.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:47:49.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:47:49.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:48:09.760 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:48:09.793 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:48:09.793 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:48:09.793 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:48:09.793 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:48:09.793 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:48:09.793 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:48:09.793 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:48:09.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:48:09.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:48:09.795 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:48:09.795 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:48:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:48:09.795 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:48:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:48:09.795 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:48:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:48:09.795 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:48:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:48:09.795 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:48:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:48:09.795 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:48:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:48:09.795 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:48:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:48:09.795 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:48:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:48:09.795 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:48:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:48:09.795 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:48:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:48:09.795 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:48:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:48:09.795 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:48:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:48:09.795 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:48:09.795 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:48:09.795 139328 ERROR oslo_service.periodic_task >2018-06-24 23:48:39.677 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:48:39.678 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:48:39.686 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:48:39.686 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:48:39.740 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:48:39.740 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:48:39.740 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:48:39.740 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:48:39.740 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:48:39.740 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:48:39.740 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:48:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:48:39.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.057 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:48:39.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:48:49.744 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:48:49.778 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:48:49.778 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:48:49.778 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:48:49.778 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:48:49.778 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:48:49.778 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:48:49.778 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:48:49.781 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:48:49.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.095 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:48:49.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:49:09.782 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:49:09.817 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:49:09.817 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:49:09.817 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:49:09.817 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:49:09.817 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:49:09.817 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:49:09.817 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:49:09.819 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:49:09.819 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.132 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:49:09.819 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:49:09.819 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:49:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:49:09.819 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:49:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:49:09.819 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:49:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:49:09.819 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:49:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:49:09.819 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:49:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:49:09.819 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:49:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:49:09.819 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:49:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:49:09.819 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:49:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:49:09.819 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:49:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:49:09.819 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:49:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:49:09.819 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:49:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:49:09.819 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:49:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:49:09.819 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:49:09.819 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:49:09.819 139328 ERROR oslo_service.periodic_task >2018-06-24 23:49:39.681 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:49:39.681 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:49:39.689 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:49:39.689 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:49:39.723 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:49:39.723 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:49:39.723 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:49:39.723 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:49:39.723 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:49:39.723 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:49:39.723 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:49:39.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:49:39.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:49:39.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:49:49.726 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:49:49.766 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:49:49.766 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:49:49.766 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:49:49.766 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:49:49.766 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:49:49.766 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:49:49.766 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:49:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:49:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:49:49.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:50:09.771 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:50:09.804 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:50:09.804 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:50:09.804 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:50:09.804 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:50:09.804 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:50:09.804 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:50:09.804 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:50:09.805 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:50:09.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:50:09.806 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:50:09.806 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:50:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:50:09.806 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:50:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:50:09.806 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:50:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:50:09.806 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:50:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:50:09.806 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:50:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:50:09.806 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:50:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:50:09.806 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:50:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:50:09.806 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:50:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:50:09.806 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:50:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:50:09.806 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:50:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:50:09.806 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:50:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:50:09.806 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:50:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:50:09.806 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:50:09.806 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:50:09.806 139328 ERROR oslo_service.periodic_task >2018-06-24 23:50:39.678 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:50:39.679 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:50:39.685 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:50:39.686 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:50:39.719 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:50:39.719 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:50:39.719 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:50:39.719 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:50:39.719 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:50:39.719 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:50:39.719 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:50:39.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:50:39.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:50:39.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:50:49.722 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:50:49.755 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:50:49.755 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:50:49.755 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:50:49.755 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:50:49.755 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:50:49.755 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:50:49.755 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:50:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:50:49.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:50:49.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:51:09.759 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:51:09.790 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:51:09.790 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:51:09.790 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:51:09.790 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:51:09.790 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:51:09.790 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:51:09.790 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:51:09.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:51:09.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.107 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:51:09.793 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:51:09.793 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:51:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:51:09.793 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:51:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:51:09.793 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:51:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:51:09.793 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:51:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:51:09.793 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:51:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:51:09.793 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:51:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:51:09.793 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:51:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:51:09.793 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:51:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:51:09.793 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:51:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:51:09.793 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:51:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:51:09.793 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:51:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:51:09.793 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:51:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:51:09.793 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:51:09.793 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:51:09.793 139328 ERROR oslo_service.periodic_task >2018-06-24 23:51:39.674 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:51:39.675 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:51:39.707 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:51:39.707 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:51:39.707 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:51:39.707 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:51:39.707 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:51:39.707 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:51:39.707 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:51:39.709 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:51:39.710 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:51:39.710 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:51:49.710 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:51:49.743 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:51:49.743 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:51:49.743 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:51:49.743 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:51:49.743 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:51:49.743 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:51:49.743 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:51:49.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:51:49.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.07 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:51:49.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:52:09.746 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:52:09.784 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:52:09.784 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:52:09.784 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:52:09.784 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:52:09.784 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:52:09.784 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:52:09.784 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:52:09.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:52:09.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:52:09.787 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:52:09.787 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:52:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:52:09.787 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:52:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:52:09.787 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:52:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:52:09.787 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:52:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:52:09.787 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:52:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:52:09.787 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:52:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:52:09.787 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:52:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:52:09.787 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:52:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:52:09.787 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:52:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:52:09.787 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:52:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:52:09.787 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:52:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:52:09.787 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:52:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:52:09.787 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:52:09.787 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:52:09.787 139328 ERROR oslo_service.periodic_task >2018-06-24 23:52:39.678 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:52:39.679 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:52:39.685 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:52:39.686 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:52:39.720 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:52:39.720 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:52:39.720 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:52:39.720 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:52:39.720 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:52:39.720 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:52:39.720 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:52:39.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:52:39.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:52:39.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:52:49.724 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:52:49.756 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:52:49.756 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:52:49.756 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:52:49.756 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:52:49.756 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:52:49.756 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:52:49.756 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:52:49.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:52:49.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:52:49.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:53:09.759 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:53:09.794 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:53:09.794 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:53:09.794 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:53:09.794 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:53:09.794 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:53:09.794 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:53:09.794 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:53:09.797 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:53:09.797 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:53:09.797 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:53:09.797 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:53:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:53:09.797 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:53:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:53:09.797 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:53:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:53:09.797 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:53:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:53:09.797 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:53:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:53:09.797 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:53:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:53:09.797 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:53:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:53:09.797 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:53:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:53:09.797 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:53:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:53:09.797 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:53:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:53:09.797 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:53:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:53:09.797 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:53:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:53:09.797 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:53:09.797 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:53:09.797 139328 ERROR oslo_service.periodic_task >2018-06-24 23:53:39.679 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:53:39.680 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:53:39.687 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:53:39.688 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:53:39.722 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:53:39.722 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:53:39.722 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:53:39.722 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:53:39.722 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:53:39.722 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:53:39.722 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:53:39.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:53:39.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:53:39.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:53:49.726 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:53:49.764 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:53:49.764 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:53:49.764 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:53:49.764 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:53:49.764 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:53:49.764 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:53:49.764 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:53:49.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:53:49.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:53:49.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:54:09.767 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:54:09.799 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:54:09.799 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:54:09.799 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:54:09.799 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:54:09.799 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:54:09.799 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:54:09.799 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:54:09.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:54:09.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:54:09.802 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:54:09.802 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:54:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:54:09.802 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:54:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:54:09.802 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:54:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:54:09.802 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:54:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:54:09.802 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:54:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:54:09.802 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:54:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:54:09.802 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:54:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:54:09.802 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:54:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:54:09.802 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:54:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:54:09.802 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:54:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:54:09.802 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:54:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:54:09.802 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:54:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:54:09.802 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:54:09.802 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:54:09.802 139328 ERROR oslo_service.periodic_task >2018-06-24 23:54:39.684 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:54:39.685 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:54:39.693 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:54:39.694 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:54:39.732 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:54:39.732 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:54:39.732 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:54:39.732 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:54:39.732 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:54:39.732 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:54:39.732 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:54:39.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:54:39.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:54:39.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:54:49.735 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:54:49.769 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:54:49.769 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:54:49.769 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:54:49.769 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:54:49.769 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:54:49.769 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:54:49.769 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:54:49.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:54:49.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:54:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:55:09.773 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:55:09.806 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:55:09.806 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:55:09.806 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:55:09.806 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:55:09.806 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:55:09.806 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:55:09.806 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:55:09.808 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:55:09.809 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:55:09.809 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:55:09.809 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:55:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:55:09.809 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:55:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:55:09.809 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:55:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:55:09.809 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:55:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:55:09.809 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:55:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:55:09.809 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:55:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:55:09.809 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:55:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:55:09.809 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:55:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:55:09.809 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:55:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:55:09.809 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:55:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:55:09.809 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:55:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:55:09.809 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:55:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:55:09.809 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:55:09.809 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:55:09.809 139328 ERROR oslo_service.periodic_task >2018-06-24 23:55:39.681 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:55:39.682 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:55:39.695 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:55:39.695 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:55:39.728 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:55:39.728 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:55:39.728 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:55:39.728 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:55:39.728 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:55:39.728 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:55:39.728 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:55:39.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:55:39.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:55:39.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:55:49.732 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:55:49.766 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:55:49.766 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:55:49.766 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:55:49.766 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:55:49.766 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:55:49.766 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:55:49.766 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:55:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:55:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:55:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:56:09.769 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:56:09.803 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:56:09.803 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:56:09.803 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:56:09.803 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:56:09.803 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:56:09.803 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:56:09.803 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:56:09.805 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:56:09.805 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:56:09.805 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:56:09.805 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:56:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:56:09.805 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:56:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:56:09.805 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:56:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:56:09.805 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:56:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:56:09.805 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:56:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:56:09.805 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:56:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:56:09.805 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:56:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:56:09.805 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:56:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:56:09.805 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:56:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:56:09.805 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:56:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:56:09.805 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:56:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:56:09.805 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:56:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:56:09.805 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:56:09.805 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:56:09.805 139328 ERROR oslo_service.periodic_task >2018-06-24 23:56:39.677 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:56:39.679 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:56:39.685 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:56:39.686 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:56:39.725 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:56:39.725 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:56:39.725 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:56:39.725 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:56:39.725 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:56:39.725 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:56:39.725 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:56:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:56:39.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.043 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:56:39.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:56:49.730 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:56:49.763 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:56:49.763 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:56:49.763 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:56:49.763 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:56:49.763 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:56:49.763 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:56:49.763 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:56:49.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:56:49.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.08 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:56:49.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:57:09.766 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:57:09.806 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:57:09.806 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:57:09.806 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:57:09.806 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:57:09.806 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:57:09.806 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:57:09.806 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:57:09.809 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:57:09.809 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.124 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:57:09.810 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:57:09.810 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:57:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:57:09.810 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:57:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:57:09.810 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:57:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:57:09.810 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:57:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:57:09.810 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:57:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:57:09.810 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:57:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:57:09.810 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:57:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:57:09.810 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:57:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:57:09.810 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:57:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:57:09.810 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:57:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:57:09.810 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:57:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:57:09.810 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:57:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:57:09.810 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:57:09.810 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:57:09.810 139328 ERROR oslo_service.periodic_task >2018-06-24 23:57:39.681 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:57:39.682 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:57:39.690 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:57:39.691 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:57:39.726 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:57:39.726 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:57:39.726 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:57:39.726 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:57:39.726 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:57:39.726 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:57:39.726 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:57:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:57:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:57:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:57:49.729 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:57:49.766 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:57:49.766 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:57:49.766 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:57:49.766 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:57:49.766 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:57:49.766 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:57:49.766 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:57:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:57:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:57:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:58:09.770 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:58:09.803 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:58:09.803 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:58:09.803 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:58:09.803 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:58:09.803 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:58:09.803 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:58:09.803 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:58:09.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:58:09.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:58:09.807 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:58:09.807 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:58:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:58:09.807 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:58:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:58:09.807 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:58:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:58:09.807 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:58:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:58:09.807 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:58:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:58:09.807 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:58:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:58:09.807 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:58:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:58:09.807 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:58:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:58:09.807 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:58:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:58:09.807 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:58:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:58:09.807 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:58:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:58:09.807 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:58:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:58:09.807 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:58:09.807 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:58:09.807 139328 ERROR oslo_service.periodic_task >2018-06-24 23:58:39.678 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:58:39.679 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:58:39.686 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:58:39.686 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:58:39.719 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:58:39.719 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:58:39.719 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:58:39.719 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:58:39.719 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:58:39.719 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:58:39.719 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:58:39.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:58:39.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:58:39.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:58:49.722 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:58:49.755 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:58:49.755 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:58:49.755 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:58:49.755 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:58:49.755 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:58:49.755 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:58:49.755 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:58:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:58:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.07 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:58:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:59:09.758 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:59:09.791 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:59:09.791 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:59:09.791 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:59:09.791 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:59:09.791 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:59:09.791 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:59:09.791 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:59:09.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:59:09.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.107 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:59:09.794 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:59:09.794 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-24 23:59:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-24 23:59:09.794 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-24 23:59:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-24 23:59:09.794 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-24 23:59:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-24 23:59:09.794 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-24 23:59:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-24 23:59:09.794 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-24 23:59:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-24 23:59:09.794 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-24 23:59:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-24 23:59:09.794 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-24 23:59:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-24 23:59:09.794 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-24 23:59:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-24 23:59:09.794 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-24 23:59:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-24 23:59:09.794 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-24 23:59:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-24 23:59:09.794 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-24 23:59:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-24 23:59:09.794 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-24 23:59:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-24 23:59:09.794 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-24 23:59:09.794 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-24 23:59:09.794 139328 ERROR oslo_service.periodic_task >2018-06-24 23:59:39.676 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:59:39.677 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-24 23:59:39.683 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-24 23:59:39.684 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:59:39.718 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:59:39.718 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:59:39.718 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:59:39.718 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:59:39.718 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:59:39.718 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:59:39.718 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:59:39.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:59:39.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:59:39.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-24 23:59:49.722 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-24 23:59:49.776 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-24 23:59:49.776 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-24 23:59:49.776 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-24 23:59:49.776 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-24 23:59:49.776 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-24 23:59:49.776 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-24 23:59:49.776 139328 ERROR cinder.volume.drivers.rbd >2018-06-24 23:59:49.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-24 23:59:49.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.095 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-24 23:59:49.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:00:09.779 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:00:09.814 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:00:09.814 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:00:09.814 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:00:09.814 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:00:09.814 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:00:09.814 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:00:09.814 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:00:09.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:00:09.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.133 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:00:09.817 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:00:09.817 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:00:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:00:09.817 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:00:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:00:09.817 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:00:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:00:09.817 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:00:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:00:09.817 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:00:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:00:09.817 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:00:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:00:09.817 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:00:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:00:09.817 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:00:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:00:09.817 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:00:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:00:09.817 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:00:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:00:09.817 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:00:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:00:09.817 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:00:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:00:09.817 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:00:09.817 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:00:09.817 139328 ERROR oslo_service.periodic_task >2018-06-25 00:00:39.678 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:00:39.680 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:00:39.687 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:00:39.687 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:00:39.721 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:00:39.721 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:00:39.721 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:00:39.721 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:00:39.721 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:00:39.721 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:00:39.721 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:00:39.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:00:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:00:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:00:49.724 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:00:49.756 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:00:49.756 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:00:49.756 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:00:49.756 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:00:49.756 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:00:49.756 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:00:49.756 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:00:49.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:00:49.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:00:49.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:01:09.760 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:01:09.812 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:01:09.812 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:01:09.812 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:01:09.812 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:01:09.812 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:01:09.812 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:01:09.812 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:01:09.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:01:09.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.127 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:01:09.816 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:01:09.816 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:01:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:01:09.816 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:01:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:01:09.816 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:01:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:01:09.816 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:01:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:01:09.816 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:01:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:01:09.816 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:01:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:01:09.816 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:01:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:01:09.816 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:01:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:01:09.816 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:01:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:01:09.816 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:01:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:01:09.816 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:01:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:01:09.816 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:01:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:01:09.816 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:01:09.816 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:01:09.816 139328 ERROR oslo_service.periodic_task >2018-06-25 00:01:39.677 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:01:39.679 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:01:39.686 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:01:39.686 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:01:39.723 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:01:39.723 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:01:39.723 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:01:39.723 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:01:39.723 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:01:39.723 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:01:39.723 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:01:39.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:01:39.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:01:39.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:01:49.726 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:01:49.758 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:01:49.758 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:01:49.758 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:01:49.758 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:01:49.758 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:01:49.758 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:01:49.758 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:01:49.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:01:49.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:01:49.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:02:09.761 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:02:09.813 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:02:09.813 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:02:09.813 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:02:09.813 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:02:09.813 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:02:09.813 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:02:09.813 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:02:09.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:02:09.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.129 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:02:09.816 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:02:09.816 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:02:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:02:09.816 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:02:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:02:09.816 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:02:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:02:09.816 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:02:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:02:09.816 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:02:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:02:09.816 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:02:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:02:09.816 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:02:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:02:09.816 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:02:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:02:09.816 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:02:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:02:09.816 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:02:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:02:09.816 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:02:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:02:09.816 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:02:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:02:09.816 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:02:09.816 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:02:09.816 139328 ERROR oslo_service.periodic_task >2018-06-25 00:02:39.677 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:02:39.678 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:02:39.686 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:02:39.687 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:02:39.722 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:02:39.722 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:02:39.722 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:02:39.722 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:02:39.722 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:02:39.722 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:02:39.722 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:02:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:02:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:02:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:02:49.725 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:02:49.756 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:02:49.756 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:02:49.756 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:02:49.756 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:02:49.756 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:02:49.756 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:02:49.756 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:02:49.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:02:49.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:02:49.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:03:09.759 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:03:09.794 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:03:09.794 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:03:09.794 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:03:09.794 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:03:09.794 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:03:09.794 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:03:09.794 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:03:09.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:03:09.797 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:03:09.797 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:03:09.797 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:03:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:03:09.797 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:03:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:03:09.797 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:03:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:03:09.797 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:03:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:03:09.797 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:03:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:03:09.797 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:03:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:03:09.797 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:03:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:03:09.797 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:03:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:03:09.797 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:03:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:03:09.797 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:03:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:03:09.797 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:03:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:03:09.797 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:03:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:03:09.797 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:03:09.797 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:03:09.797 139328 ERROR oslo_service.periodic_task >2018-06-25 00:03:39.679 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:03:39.681 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:03:39.687 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:03:39.687 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:03:39.724 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:03:39.724 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:03:39.724 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:03:39.724 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:03:39.724 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:03:39.724 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:03:39.724 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:03:39.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:03:39.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:03:39.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:03:49.729 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:03:49.764 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:03:49.764 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:03:49.764 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:03:49.764 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:03:49.764 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:03:49.764 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:03:49.764 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:03:49.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:03:49.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:03:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:04:09.768 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:04:09.804 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:04:09.804 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:04:09.804 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:04:09.804 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:04:09.804 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:04:09.804 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:04:09.804 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:04:09.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:04:09.807 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.119 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:04:09.807 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:04:09.807 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:04:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:04:09.807 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:04:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:04:09.807 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:04:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:04:09.807 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:04:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:04:09.807 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:04:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:04:09.807 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:04:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:04:09.807 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:04:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:04:09.807 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:04:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:04:09.807 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:04:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:04:09.807 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:04:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:04:09.807 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:04:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:04:09.807 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:04:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:04:09.807 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:04:09.807 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:04:09.807 139328 ERROR oslo_service.periodic_task >2018-06-25 00:04:39.679 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:04:39.680 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:04:39.688 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:04:39.689 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:04:39.726 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:04:39.726 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:04:39.726 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:04:39.726 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:04:39.726 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:04:39.726 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:04:39.726 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:04:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:04:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:04:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:04:49.729 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:04:49.762 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:04:49.762 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:04:49.762 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:04:49.762 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:04:49.762 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:04:49.762 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:04:49.762 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:04:49.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:04:49.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:04:49.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:05:09.766 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:05:09.800 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:05:09.800 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:05:09.800 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:05:09.800 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:05:09.800 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:05:09.800 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:05:09.800 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:05:09.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:05:09.803 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:05:09.803 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:05:09.803 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:05:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:05:09.803 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:05:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:05:09.803 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:05:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:05:09.803 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:05:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:05:09.803 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:05:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:05:09.803 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:05:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:05:09.803 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:05:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:05:09.803 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:05:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:05:09.803 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:05:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:05:09.803 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:05:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:05:09.803 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:05:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:05:09.803 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:05:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:05:09.803 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:05:09.803 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:05:09.803 139328 ERROR oslo_service.periodic_task >2018-06-25 00:05:39.675 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:05:39.676 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:05:39.685 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:05:39.686 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:05:39.721 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:05:39.721 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:05:39.721 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:05:39.721 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:05:39.721 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:05:39.721 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:05:39.721 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:05:39.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:05:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:05:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:05:49.725 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:05:49.760 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:05:49.760 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:05:49.760 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:05:49.760 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:05:49.760 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:05:49.760 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:05:49.760 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:05:49.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:05:49.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:05:49.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:06:09.763 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:06:09.794 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:06:09.794 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:06:09.794 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:06:09.794 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:06:09.794 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:06:09.794 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:06:09.794 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:06:09.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:06:09.797 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:06:09.797 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:06:09.797 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:06:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:06:09.797 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:06:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:06:09.797 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:06:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:06:09.797 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:06:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:06:09.797 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:06:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:06:09.797 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:06:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:06:09.797 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:06:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:06:09.797 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:06:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:06:09.797 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:06:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:06:09.797 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:06:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:06:09.797 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:06:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:06:09.797 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:06:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:06:09.797 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:06:09.797 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:06:09.797 139328 ERROR oslo_service.periodic_task >2018-06-25 00:06:39.679 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:06:39.680 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:06:39.687 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:06:39.687 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:06:39.721 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:06:39.721 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:06:39.721 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:06:39.721 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:06:39.721 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:06:39.721 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:06:39.721 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:06:39.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:06:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:06:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:06:49.725 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:06:49.756 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:06:49.756 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:06:49.756 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:06:49.756 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:06:49.756 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:06:49.756 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:06:49.756 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:06:49.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:06:49.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:06:49.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:07:09.760 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:07:09.794 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:07:09.794 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:07:09.794 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:07:09.794 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:07:09.794 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:07:09.794 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:07:09.794 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:07:09.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:07:09.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:07:09.796 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:07:09.796 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:07:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:07:09.796 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:07:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:07:09.796 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:07:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:07:09.796 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:07:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:07:09.796 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:07:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:07:09.796 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:07:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:07:09.796 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:07:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:07:09.796 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:07:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:07:09.796 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:07:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:07:09.796 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:07:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:07:09.796 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:07:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:07:09.796 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:07:09.796 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:07:09.796 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:07:09.796 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:07:09.796 139328 ERROR oslo_service.periodic_task >2018-06-25 00:07:39.678 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:07:39.678 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:07:39.687 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:07:39.687 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:07:39.720 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:07:39.720 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:07:39.720 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:07:39.720 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:07:39.720 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:07:39.720 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:07:39.720 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:07:39.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:07:39.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:07:39.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:07:49.723 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:07:49.758 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:07:49.758 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:07:49.758 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:07:49.758 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:07:49.758 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:07:49.758 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:07:49.758 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:07:49.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:07:49.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:07:49.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:08:09.761 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:08:09.813 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:08:09.813 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:08:09.813 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:08:09.813 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:08:09.813 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:08:09.813 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:08:09.813 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:08:09.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:08:09.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.128 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:08:09.816 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:08:09.816 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:08:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:08:09.816 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:08:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:08:09.816 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:08:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:08:09.816 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:08:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:08:09.816 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:08:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:08:09.816 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:08:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:08:09.816 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:08:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:08:09.816 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:08:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:08:09.816 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:08:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:08:09.816 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:08:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:08:09.816 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:08:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:08:09.816 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:08:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:08:09.816 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:08:09.816 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:08:09.816 139328 ERROR oslo_service.periodic_task >2018-06-25 00:08:39.678 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:08:39.679 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:08:39.685 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:08:39.686 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:08:39.720 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:08:39.720 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:08:39.720 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:08:39.720 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:08:39.720 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:08:39.720 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:08:39.720 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:08:39.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:08:39.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:08:39.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:08:49.724 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:08:49.756 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:08:49.756 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:08:49.756 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:08:49.756 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:08:49.756 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:08:49.756 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:08:49.756 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:08:49.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:08:49.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:08:49.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:09:09.760 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:09:09.796 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:09:09.796 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:09:09.796 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:09:09.796 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:09:09.796 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:09:09.796 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:09:09.796 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:09:09.798 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:09:09.798 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:09:09.798 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:09:09.798 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:09:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:09:09.798 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:09:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:09:09.798 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:09:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:09:09.798 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:09:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:09:09.798 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:09:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:09:09.798 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:09:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:09:09.798 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:09:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:09:09.798 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:09:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:09:09.798 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:09:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:09:09.798 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:09:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:09:09.798 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:09:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:09:09.798 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:09:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:09:09.798 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:09:09.798 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:09:09.798 139328 ERROR oslo_service.periodic_task >2018-06-25 00:09:39.680 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:09:39.681 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:09:39.688 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:09:39.688 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:09:39.720 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:09:39.720 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:09:39.720 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:09:39.720 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:09:39.720 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:09:39.720 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:09:39.720 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:09:39.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:09:39.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:09:39.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:09:49.723 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:09:49.756 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:09:49.756 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:09:49.756 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:09:49.756 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:09:49.756 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:09:49.756 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:09:49.756 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:09:49.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:09:49.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.07 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:09:49.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:10:09.760 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:10:09.795 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:10:09.795 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:10:09.795 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:10:09.795 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:10:09.795 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:10:09.795 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:10:09.795 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:10:09.798 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:10:09.798 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:10:09.799 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:10:09.799 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:10:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:10:09.799 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:10:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:10:09.799 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:10:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:10:09.799 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:10:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:10:09.799 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:10:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:10:09.799 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:10:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:10:09.799 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:10:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:10:09.799 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:10:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:10:09.799 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:10:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:10:09.799 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:10:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:10:09.799 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:10:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:10:09.799 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:10:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:10:09.799 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:10:09.799 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:10:09.799 139328 ERROR oslo_service.periodic_task >2018-06-25 00:10:39.681 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:10:39.682 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:10:39.689 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:10:39.689 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:10:39.730 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:10:39.730 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:10:39.730 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:10:39.730 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:10:39.730 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:10:39.730 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:10:39.730 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:10:39.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:10:39.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.043 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:10:39.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:10:49.733 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:10:49.765 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:10:49.765 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:10:49.765 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:10:49.765 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:10:49.765 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:10:49.765 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:10:49.765 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:10:49.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:10:49.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:10:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:11:09.768 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:11:09.818 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:11:09.818 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:11:09.818 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:11:09.818 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:11:09.818 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:11:09.818 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:11:09.818 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:11:09.820 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:11:09.820 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.13 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:11:09.821 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:11:09.821 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:11:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:11:09.821 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:11:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:11:09.821 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:11:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:11:09.821 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:11:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:11:09.821 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:11:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:11:09.821 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:11:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:11:09.821 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:11:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:11:09.821 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:11:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:11:09.821 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:11:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:11:09.821 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:11:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:11:09.821 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:11:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:11:09.821 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:11:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:11:09.821 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:11:09.821 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:11:09.821 139328 ERROR oslo_service.periodic_task >2018-06-25 00:11:39.682 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:11:39.684 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:11:39.690 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:11:39.691 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:11:39.727 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:11:39.727 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:11:39.727 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:11:39.727 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:11:39.727 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:11:39.727 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:11:39.727 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:11:39.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:11:39.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:11:39.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:11:49.732 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:11:49.763 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:11:49.763 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:11:49.763 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:11:49.763 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:11:49.763 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:11:49.763 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:11:49.763 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:11:49.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:11:49.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:11:49.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:12:09.766 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:12:09.800 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:12:09.800 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:12:09.800 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:12:09.800 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:12:09.800 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:12:09.800 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:12:09.800 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:12:09.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:12:09.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:12:09.803 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:12:09.803 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:12:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:12:09.803 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:12:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:12:09.803 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:12:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:12:09.803 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:12:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:12:09.803 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:12:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:12:09.803 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:12:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:12:09.803 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:12:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:12:09.803 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:12:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:12:09.803 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:12:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:12:09.803 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:12:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:12:09.803 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:12:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:12:09.803 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:12:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:12:09.803 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:12:09.803 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:12:09.803 139328 ERROR oslo_service.periodic_task >2018-06-25 00:12:39.684 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:12:39.685 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:12:39.700 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:12:39.701 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:12:39.740 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:12:39.740 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:12:39.740 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:12:39.740 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:12:39.740 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:12:39.740 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:12:39.740 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:12:39.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:12:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.041 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:12:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:12:49.744 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:12:49.776 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:12:49.776 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:12:49.776 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:12:49.776 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:12:49.776 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:12:49.776 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:12:49.776 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:12:49.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:12:49.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:12:49.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:13:09.779 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:13:09.812 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:13:09.812 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:13:09.812 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:13:09.812 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:13:09.812 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:13:09.812 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:13:09.812 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:13:09.814 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:13:09.814 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:13:09.814 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:13:09.814 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:13:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:13:09.814 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:13:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:13:09.814 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:13:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:13:09.814 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:13:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:13:09.814 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:13:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:13:09.814 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:13:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:13:09.814 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:13:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:13:09.814 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:13:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:13:09.814 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:13:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:13:09.814 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:13:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:13:09.814 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:13:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:13:09.814 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:13:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:13:09.814 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:13:09.814 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:13:09.814 139328 ERROR oslo_service.periodic_task >2018-06-25 00:13:39.686 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:13:39.687 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:13:39.695 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:13:39.695 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:13:39.729 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:13:39.729 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:13:39.729 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:13:39.729 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:13:39.729 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:13:39.729 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:13:39.729 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:13:39.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:13:39.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:13:39.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:13:49.733 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:13:49.770 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:13:49.770 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:13:49.770 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:13:49.770 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:13:49.770 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:13:49.770 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:13:49.770 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:13:49.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:13:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:13:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:14:09.774 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:14:09.809 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:14:09.809 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:14:09.809 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:14:09.809 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:14:09.809 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:14:09.809 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:14:09.809 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:14:09.812 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:14:09.813 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:14:09.813 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:14:09.813 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:14:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:14:09.813 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:14:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:14:09.813 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:14:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:14:09.813 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:14:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:14:09.813 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:14:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:14:09.813 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:14:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:14:09.813 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:14:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:14:09.813 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:14:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:14:09.813 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:14:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:14:09.813 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:14:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:14:09.813 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:14:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:14:09.813 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:14:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:14:09.813 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:14:09.813 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:14:09.813 139328 ERROR oslo_service.periodic_task >2018-06-25 00:14:39.685 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:14:39.686 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:14:39.693 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:14:39.693 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:14:39.726 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:14:39.726 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:14:39.726 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:14:39.726 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:14:39.726 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:14:39.726 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:14:39.726 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:14:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:14:39.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:14:39.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:14:49.729 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:14:49.763 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:14:49.763 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:14:49.763 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:14:49.763 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:14:49.763 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:14:49.763 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:14:49.763 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:14:49.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:14:49.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:14:49.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:15:09.766 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:15:09.799 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:15:09.799 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:15:09.799 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:15:09.799 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:15:09.799 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:15:09.799 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:15:09.799 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:15:09.801 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:15:09.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:15:09.802 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:15:09.802 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:15:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:15:09.802 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:15:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:15:09.802 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:15:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:15:09.802 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:15:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:15:09.802 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:15:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:15:09.802 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:15:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:15:09.802 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:15:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:15:09.802 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:15:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:15:09.802 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:15:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:15:09.802 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:15:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:15:09.802 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:15:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:15:09.802 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:15:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:15:09.802 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:15:09.802 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:15:09.802 139328 ERROR oslo_service.periodic_task >2018-06-25 00:15:39.684 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:15:39.684 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:15:39.692 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:15:39.693 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:15:39.727 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:15:39.727 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:15:39.727 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:15:39.727 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:15:39.727 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:15:39.727 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:15:39.727 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:15:39.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:15:39.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:15:39.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:15:49.730 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:15:49.764 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:15:49.764 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:15:49.764 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:15:49.764 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:15:49.764 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:15:49.764 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:15:49.764 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:15:49.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:15:49.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:15:49.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:16:09.767 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:16:09.820 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:16:09.820 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:16:09.820 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:16:09.820 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:16:09.820 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:16:09.820 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:16:09.820 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:16:09.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:16:09.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.13 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:16:09.824 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:16:09.824 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:16:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:16:09.824 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:16:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:16:09.824 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:16:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:16:09.824 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:16:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:16:09.824 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:16:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:16:09.824 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:16:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:16:09.824 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:16:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:16:09.824 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:16:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:16:09.824 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:16:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:16:09.824 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:16:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:16:09.824 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:16:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:16:09.824 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:16:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:16:09.824 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:16:09.824 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:16:09.824 139328 ERROR oslo_service.periodic_task >2018-06-25 00:16:39.685 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:16:39.687 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:16:39.694 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:16:39.695 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:16:39.728 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:16:39.728 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:16:39.728 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:16:39.728 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:16:39.728 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:16:39.728 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:16:39.728 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:16:39.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:16:39.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:16:39.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:16:49.731 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:16:49.764 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:16:49.764 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:16:49.764 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:16:49.764 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:16:49.764 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:16:49.764 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:16:49.764 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:16:49.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:16:49.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:16:49.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:17:09.768 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:17:09.804 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:17:09.804 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:17:09.804 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:17:09.804 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:17:09.804 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:17:09.804 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:17:09.804 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:17:09.807 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:17:09.807 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:17:09.808 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:17:09.808 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:17:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:17:09.808 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:17:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:17:09.808 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:17:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:17:09.808 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:17:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:17:09.808 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:17:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:17:09.808 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:17:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:17:09.808 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:17:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:17:09.808 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:17:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:17:09.808 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:17:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:17:09.808 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:17:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:17:09.808 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:17:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:17:09.808 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:17:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:17:09.808 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:17:09.808 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:17:09.808 139328 ERROR oslo_service.periodic_task >2018-06-25 00:17:39.689 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:17:39.691 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:17:39.697 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:17:39.698 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:17:39.735 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:17:39.735 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:17:39.735 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:17:39.735 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:17:39.735 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:17:39.735 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:17:39.735 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:17:39.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:17:39.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.041 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:17:39.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:17:49.739 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:17:49.775 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:17:49.775 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:17:49.775 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:17:49.775 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:17:49.775 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:17:49.775 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:17:49.775 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:17:49.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:17:49.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.08 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:17:49.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:18:09.778 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:18:09.814 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:18:09.814 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:18:09.814 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:18:09.814 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:18:09.814 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:18:09.814 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:18:09.814 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:18:09.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:18:09.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.118 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:18:09.817 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:18:09.817 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:18:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:18:09.817 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:18:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:18:09.817 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:18:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:18:09.817 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:18:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:18:09.817 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:18:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:18:09.817 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:18:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:18:09.817 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:18:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:18:09.817 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:18:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:18:09.817 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:18:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:18:09.817 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:18:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:18:09.817 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:18:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:18:09.817 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:18:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:18:09.817 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:18:09.817 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:18:09.817 139328 ERROR oslo_service.periodic_task >2018-06-25 00:18:39.688 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:18:39.689 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:18:39.696 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:18:39.697 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:18:39.731 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:18:39.731 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:18:39.731 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:18:39.731 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:18:39.731 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:18:39.731 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:18:39.731 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:18:39.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:18:39.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:18:39.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:18:49.734 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:18:49.769 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:18:49.769 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:18:49.769 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:18:49.769 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:18:49.769 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:18:49.769 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:18:49.769 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:18:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:18:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:18:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:19:09.775 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:19:09.808 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:19:09.808 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:19:09.808 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:19:09.808 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:19:09.808 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:19:09.808 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:19:09.808 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:19:09.811 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:19:09.811 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:19:09.811 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:19:09.811 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:19:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:19:09.811 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:19:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:19:09.811 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:19:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:19:09.811 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:19:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:19:09.811 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:19:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:19:09.811 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:19:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:19:09.811 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:19:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:19:09.811 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:19:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:19:09.811 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:19:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:19:09.811 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:19:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:19:09.811 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:19:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:19:09.811 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:19:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:19:09.811 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:19:09.811 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:19:09.811 139328 ERROR oslo_service.periodic_task >2018-06-25 00:19:39.693 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:19:39.694 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:19:39.701 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:19:39.702 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:19:39.734 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:19:39.734 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:19:39.734 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:19:39.734 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:19:39.734 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:19:39.734 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:19:39.734 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:19:39.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:19:39.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:19:39.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:19:49.738 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:19:49.775 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:19:49.775 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:19:49.775 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:19:49.775 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:19:49.775 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:19:49.775 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:19:49.775 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:19:49.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:19:49.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:19:49.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:20:09.778 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:20:09.814 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:20:09.814 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:20:09.814 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:20:09.814 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:20:09.814 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:20:09.814 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:20:09.814 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:20:09.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:20:09.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:20:09.816 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:20:09.816 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:20:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:20:09.816 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:20:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:20:09.816 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:20:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:20:09.816 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:20:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:20:09.816 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:20:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:20:09.816 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:20:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:20:09.816 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:20:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:20:09.816 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:20:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:20:09.816 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:20:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:20:09.816 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:20:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:20:09.816 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:20:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:20:09.816 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:20:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:20:09.816 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:20:09.816 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:20:09.816 139328 ERROR oslo_service.periodic_task >2018-06-25 00:20:39.698 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:20:39.698 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:20:39.705 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:20:39.706 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:20:39.738 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:20:39.738 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:20:39.738 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:20:39.738 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:20:39.738 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:20:39.738 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:20:39.738 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:20:39.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:20:39.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:20:39.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:20:49.741 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:20:49.775 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:20:49.775 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:20:49.775 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:20:49.775 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:20:49.775 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:20:49.775 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:20:49.775 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:20:49.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:20:49.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:20:49.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:21:09.778 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:21:09.810 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:21:09.810 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:21:09.810 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:21:09.810 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:21:09.810 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:21:09.810 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:21:09.810 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:21:09.812 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:21:09.812 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.106 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:21:09.812 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:21:09.812 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:21:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:21:09.812 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:21:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:21:09.812 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:21:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:21:09.812 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:21:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:21:09.812 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:21:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:21:09.812 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:21:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:21:09.812 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:21:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:21:09.812 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:21:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:21:09.812 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:21:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:21:09.812 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:21:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:21:09.812 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:21:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:21:09.812 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:21:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:21:09.812 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:21:09.812 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:21:09.812 139328 ERROR oslo_service.periodic_task >2018-06-25 00:21:39.694 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:21:39.695 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:21:39.703 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:21:39.704 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:21:39.754 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:21:39.754 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:21:39.754 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:21:39.754 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:21:39.754 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:21:39.754 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:21:39.754 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:21:39.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:21:39.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.053 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:21:39.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:21:49.758 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:21:49.791 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:21:49.791 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:21:49.791 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:21:49.791 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:21:49.791 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:21:49.791 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:21:49.791 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:21:49.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:21:49.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.09 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:21:49.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:22:09.795 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:22:09.826 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:22:09.826 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:22:09.826 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:22:09.826 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:22:09.826 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:22:09.826 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:22:09.826 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:22:09.828 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:22:09.828 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.124 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:22:09.828 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:22:09.828 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:22:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:22:09.828 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:22:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:22:09.828 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:22:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:22:09.828 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:22:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:22:09.828 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:22:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:22:09.828 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:22:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:22:09.828 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:22:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:22:09.828 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:22:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:22:09.828 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:22:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:22:09.828 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:22:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:22:09.828 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:22:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:22:09.828 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:22:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:22:09.828 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:22:09.828 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:22:09.828 139328 ERROR oslo_service.periodic_task >2018-06-25 00:22:39.690 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:22:39.691 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:22:39.698 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:22:39.699 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:22:39.734 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:22:39.734 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:22:39.734 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:22:39.734 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:22:39.734 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:22:39.734 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:22:39.734 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:22:39.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:22:39.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:22:39.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:22:49.737 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:22:49.772 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:22:49.772 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:22:49.772 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:22:49.772 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:22:49.772 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:22:49.772 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:22:49.772 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:22:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:22:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:22:49.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:23:09.775 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:23:09.811 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:23:09.811 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:23:09.811 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:23:09.811 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:23:09.811 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:23:09.811 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:23:09.811 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:23:09.813 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:23:09.814 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:23:09.814 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:23:09.814 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:23:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:23:09.814 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:23:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:23:09.814 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:23:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:23:09.814 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:23:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:23:09.814 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:23:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:23:09.814 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:23:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:23:09.814 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:23:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:23:09.814 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:23:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:23:09.814 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:23:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:23:09.814 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:23:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:23:09.814 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:23:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:23:09.814 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:23:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:23:09.814 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:23:09.814 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:23:09.814 139328 ERROR oslo_service.periodic_task >2018-06-25 00:23:39.685 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:23:39.687 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:23:39.694 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:23:39.694 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:23:39.727 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:23:39.727 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:23:39.727 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:23:39.727 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:23:39.727 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:23:39.727 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:23:39.727 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:23:39.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:23:39.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:23:39.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:23:49.731 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:23:49.764 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:23:49.764 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:23:49.764 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:23:49.764 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:23:49.764 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:23:49.764 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:23:49.764 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:23:49.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:23:49.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:23:49.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:24:09.767 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:24:09.800 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:24:09.800 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:24:09.800 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:24:09.800 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:24:09.800 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:24:09.800 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:24:09.800 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:24:09.803 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:24:09.804 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:24:09.804 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:24:09.804 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:24:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:24:09.804 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:24:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:24:09.804 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:24:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:24:09.804 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:24:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:24:09.804 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:24:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:24:09.804 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:24:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:24:09.804 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:24:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:24:09.804 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:24:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:24:09.804 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:24:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:24:09.804 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:24:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:24:09.804 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:24:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:24:09.804 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:24:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:24:09.804 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:24:09.804 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:24:09.804 139328 ERROR oslo_service.periodic_task >2018-06-25 00:24:39.686 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:24:39.687 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:24:39.694 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:24:39.695 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:24:39.732 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:24:39.732 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:24:39.732 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:24:39.732 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:24:39.732 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:24:39.732 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:24:39.732 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:24:39.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:24:39.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:24:39.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:24:49.736 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:24:49.771 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:24:49.771 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:24:49.771 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:24:49.771 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:24:49.771 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:24:49.771 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:24:49.771 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:24:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:24:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:24:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:25:09.774 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:25:09.807 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:25:09.807 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:25:09.807 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:25:09.807 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:25:09.807 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:25:09.807 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:25:09.807 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:25:09.808 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:25:09.809 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:25:09.809 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:25:09.809 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:25:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:25:09.809 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:25:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:25:09.809 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:25:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:25:09.809 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:25:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:25:09.809 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:25:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:25:09.809 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:25:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:25:09.809 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:25:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:25:09.809 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:25:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:25:09.809 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:25:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:25:09.809 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:25:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:25:09.809 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:25:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:25:09.809 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:25:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:25:09.809 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:25:09.809 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:25:09.809 139328 ERROR oslo_service.periodic_task >2018-06-25 00:25:39.691 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:25:39.692 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:25:39.705 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:25:39.706 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:25:39.739 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:25:39.739 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:25:39.739 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:25:39.739 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:25:39.739 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:25:39.739 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:25:39.739 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:25:39.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:25:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:25:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:25:49.744 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:25:49.776 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:25:49.776 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:25:49.776 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:25:49.776 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:25:49.776 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:25:49.776 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:25:49.776 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:25:49.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:25:49.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:25:49.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:26:09.779 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:26:09.813 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:26:09.813 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:26:09.813 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:26:09.813 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:26:09.813 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:26:09.813 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:26:09.813 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:26:09.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:26:09.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:26:09.815 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:26:09.815 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:26:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:26:09.815 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:26:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:26:09.815 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:26:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:26:09.815 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:26:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:26:09.815 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:26:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:26:09.815 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:26:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:26:09.815 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:26:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:26:09.815 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:26:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:26:09.815 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:26:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:26:09.815 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:26:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:26:09.815 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:26:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:26:09.815 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:26:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:26:09.815 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:26:09.815 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:26:09.815 139328 ERROR oslo_service.periodic_task >2018-06-25 00:26:39.687 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:26:39.688 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:26:39.696 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:26:39.696 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:26:39.729 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:26:39.729 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:26:39.729 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:26:39.729 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:26:39.729 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:26:39.729 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:26:39.729 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:26:39.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:26:39.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:26:39.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:26:49.733 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:26:49.766 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:26:49.766 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:26:49.766 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:26:49.766 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:26:49.766 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:26:49.766 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:26:49.766 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:26:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:26:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:26:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:27:09.769 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:27:09.803 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:27:09.803 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:27:09.803 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:27:09.803 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:27:09.803 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:27:09.803 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:27:09.803 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:27:09.805 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:27:09.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:27:09.806 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:27:09.806 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:27:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:27:09.806 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:27:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:27:09.806 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:27:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:27:09.806 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:27:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:27:09.806 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:27:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:27:09.806 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:27:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:27:09.806 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:27:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:27:09.806 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:27:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:27:09.806 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:27:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:27:09.806 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:27:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:27:09.806 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:27:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:27:09.806 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:27:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:27:09.806 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:27:09.806 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:27:09.806 139328 ERROR oslo_service.periodic_task >2018-06-25 00:27:39.687 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:27:39.688 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:27:39.696 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:27:39.697 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:27:39.733 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:27:39.733 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:27:39.733 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:27:39.733 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:27:39.733 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:27:39.733 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:27:39.733 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:27:39.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:27:39.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:27:39.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:27:49.736 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:27:49.771 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:27:49.771 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:27:49.771 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:27:49.771 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:27:49.771 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:27:49.771 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:27:49.771 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:27:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:27:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:27:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:28:09.774 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:28:09.811 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:28:09.811 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:28:09.811 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:28:09.811 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:28:09.811 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:28:09.811 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:28:09.811 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:28:09.813 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:28:09.813 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:28:09.814 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:28:09.814 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:28:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:28:09.814 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:28:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:28:09.814 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:28:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:28:09.814 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:28:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:28:09.814 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:28:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:28:09.814 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:28:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:28:09.814 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:28:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:28:09.814 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:28:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:28:09.814 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:28:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:28:09.814 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:28:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:28:09.814 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:28:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:28:09.814 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:28:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:28:09.814 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:28:09.814 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:28:09.814 139328 ERROR oslo_service.periodic_task >2018-06-25 00:28:39.685 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:28:39.686 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:28:39.693 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:28:39.693 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:28:39.727 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:28:39.727 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:28:39.727 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:28:39.727 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:28:39.727 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:28:39.727 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:28:39.727 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:28:39.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:28:39.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:28:39.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:28:49.731 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:28:49.766 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:28:49.766 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:28:49.766 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:28:49.766 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:28:49.766 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:28:49.766 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:28:49.766 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:28:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:28:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:28:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:29:09.769 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:29:09.802 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:29:09.802 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:29:09.802 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:29:09.802 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:29:09.802 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:29:09.802 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:29:09.802 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:29:09.805 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:29:09.805 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:29:09.805 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:29:09.805 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:29:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:29:09.805 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:29:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:29:09.805 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:29:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:29:09.805 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:29:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:29:09.805 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:29:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:29:09.805 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:29:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:29:09.805 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:29:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:29:09.805 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:29:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:29:09.805 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:29:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:29:09.805 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:29:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:29:09.805 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:29:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:29:09.805 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:29:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:29:09.805 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:29:09.805 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:29:09.805 139328 ERROR oslo_service.periodic_task >2018-06-25 00:29:39.687 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:29:39.688 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:29:39.694 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:29:39.695 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:29:39.731 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:29:39.731 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:29:39.731 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:29:39.731 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:29:39.731 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:29:39.731 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:29:39.731 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:29:39.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:29:39.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:29:39.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:29:49.734 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:29:49.767 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:29:49.767 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:29:49.767 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:29:49.767 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:29:49.767 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:29:49.767 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:29:49.767 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:29:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:29:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:29:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:30:09.770 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:30:09.803 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:30:09.803 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:30:09.803 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:30:09.803 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:30:09.803 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:30:09.803 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:30:09.803 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:30:09.805 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:30:09.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:30:09.806 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:30:09.806 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:30:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:30:09.806 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:30:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:30:09.806 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:30:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:30:09.806 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:30:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:30:09.806 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:30:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:30:09.806 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:30:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:30:09.806 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:30:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:30:09.806 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:30:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:30:09.806 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:30:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:30:09.806 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:30:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:30:09.806 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:30:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:30:09.806 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:30:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:30:09.806 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:30:09.806 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:30:09.806 139328 ERROR oslo_service.periodic_task >2018-06-25 00:30:39.687 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:30:39.688 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:30:39.695 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:30:39.696 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:30:39.732 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:30:39.732 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:30:39.732 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:30:39.732 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:30:39.732 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:30:39.732 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:30:39.732 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:30:39.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:30:39.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:30:39.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:30:49.735 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:30:49.768 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:30:49.768 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:30:49.768 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:30:49.768 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:30:49.768 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:30:49.768 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:30:49.768 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:30:49.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:30:49.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:30:49.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:31:09.771 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:31:09.804 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:31:09.804 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:31:09.804 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:31:09.804 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:31:09.804 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:31:09.804 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:31:09.804 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:31:09.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:31:09.807 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:31:09.807 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:31:09.807 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:31:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:31:09.807 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:31:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:31:09.807 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:31:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:31:09.807 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:31:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:31:09.807 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:31:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:31:09.807 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:31:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:31:09.807 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:31:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:31:09.807 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:31:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:31:09.807 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:31:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:31:09.807 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:31:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:31:09.807 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:31:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:31:09.807 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:31:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:31:09.807 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:31:09.807 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:31:09.807 139328 ERROR oslo_service.periodic_task >2018-06-25 00:31:39.689 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:31:39.690 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:31:39.699 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:31:39.699 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:31:39.756 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:31:39.756 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:31:39.756 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:31:39.756 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:31:39.756 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:31:39.756 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:31:39.756 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:31:39.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:31:39.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.059 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:31:39.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:31:49.760 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:31:49.794 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:31:49.794 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:31:49.794 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:31:49.794 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:31:49.794 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:31:49.794 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:31:49.794 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:31:49.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:31:49.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.096 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:31:49.797 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:32:09.797 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:32:09.830 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:32:09.830 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:32:09.830 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:32:09.830 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:32:09.830 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:32:09.830 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:32:09.830 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:32:09.833 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:32:09.833 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.133 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:32:09.833 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:32:09.833 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:32:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:32:09.833 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:32:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:32:09.833 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:32:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:32:09.833 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:32:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:32:09.833 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:32:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:32:09.833 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:32:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:32:09.833 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:32:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:32:09.833 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:32:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:32:09.833 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:32:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:32:09.833 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:32:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:32:09.833 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:32:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:32:09.833 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:32:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:32:09.833 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:32:09.833 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:32:09.833 139328 ERROR oslo_service.periodic_task >2018-06-25 00:32:39.685 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:32:39.686 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:33:39.686 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:33:39.687 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:33:39.694 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:33:39.695 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:33:39.734 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:33:39.734 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:33:39.734 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:33:39.734 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:33:39.734 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:33:39.734 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:33:39.734 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:33:39.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:33:39.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.042 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:33:39.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:33:49.738 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:33:49.772 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:33:49.772 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:33:49.772 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:33:49.772 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:33:49.772 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:33:49.772 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:33:49.772 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:33:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:33:49.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.08 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:33:49.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:34:09.775 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:34:09.812 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:34:09.812 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:34:09.812 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:34:09.812 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:34:09.812 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:34:09.812 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:34:09.812 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:34:09.814 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:34:09.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.12 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:34:09.815 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:34:09.815 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:34:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:34:09.815 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:34:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:34:09.815 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:34:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:34:09.815 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:34:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:34:09.815 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:34:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:34:09.815 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:34:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:34:09.815 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:34:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:34:09.815 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:34:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:34:09.815 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:34:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:34:09.815 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:34:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:34:09.815 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:34:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:34:09.815 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:34:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:34:09.815 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:34:09.815 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:34:09.815 139328 ERROR oslo_service.periodic_task >2018-06-25 00:34:39.687 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:34:39.688 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:34:39.697 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:34:39.698 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:34:39.736 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:34:39.736 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:34:39.736 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:34:39.736 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:34:39.736 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:34:39.736 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:34:39.736 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:34:39.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:34:39.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:34:39.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:34:49.740 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:34:49.772 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:34:49.772 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:34:49.772 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:34:49.772 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:34:49.772 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:34:49.772 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:34:49.772 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:34:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:34:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:34:49.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:35:09.775 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:35:09.810 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:35:09.810 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:35:09.810 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:35:09.810 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:35:09.810 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:35:09.810 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:35:09.810 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:35:09.812 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:35:09.812 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:35:09.812 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:35:09.812 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:35:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:35:09.812 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:35:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:35:09.812 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:35:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:35:09.812 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:35:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:35:09.812 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:35:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:35:09.812 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:35:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:35:09.812 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:35:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:35:09.812 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:35:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:35:09.812 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:35:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:35:09.812 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:35:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:35:09.812 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:35:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:35:09.812 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:35:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:35:09.812 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:35:09.812 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:35:09.812 139328 ERROR oslo_service.periodic_task >2018-06-25 00:35:39.684 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:35:39.685 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:35:39.693 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:35:39.694 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:35:39.729 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:35:39.729 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:35:39.729 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:35:39.729 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:35:39.729 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:35:39.729 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:35:39.729 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:35:39.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:35:39.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:35:39.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:35:49.732 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:35:49.767 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:35:49.767 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:35:49.767 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:35:49.767 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:35:49.767 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:35:49.767 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:35:49.767 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:35:49.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:35:49.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:35:49.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:36:09.773 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:36:09.806 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:36:09.806 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:36:09.806 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:36:09.806 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:36:09.806 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:36:09.806 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:36:09.806 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:36:09.808 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:36:09.809 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:36:09.809 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:36:09.809 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:36:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:36:09.809 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:36:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:36:09.809 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:36:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:36:09.809 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:36:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:36:09.809 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:36:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:36:09.809 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:36:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:36:09.809 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:36:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:36:09.809 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:36:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:36:09.809 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:36:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:36:09.809 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:36:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:36:09.809 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:36:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:36:09.809 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:36:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:36:09.809 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:36:09.809 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:36:09.809 139328 ERROR oslo_service.periodic_task >2018-06-25 00:36:39.681 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:36:39.682 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:36:39.696 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:36:39.697 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:36:39.731 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:36:39.731 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:36:39.731 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:36:39.731 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:36:39.731 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:36:39.731 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:36:39.731 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:36:39.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:36:39.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:36:39.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:36:49.735 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:36:49.770 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:36:49.770 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:36:49.770 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:36:49.770 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:36:49.770 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:36:49.770 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:36:49.770 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:36:49.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:36:49.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:36:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:37:09.773 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:37:09.806 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:37:09.806 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:37:09.806 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:37:09.806 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:37:09.806 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:37:09.806 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:37:09.806 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:37:09.808 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:37:09.809 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:37:09.809 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:37:09.809 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:37:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:37:09.809 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:37:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:37:09.809 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:37:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:37:09.809 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:37:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:37:09.809 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:37:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:37:09.809 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:37:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:37:09.809 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:37:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:37:09.809 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:37:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:37:09.809 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:37:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:37:09.809 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:37:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:37:09.809 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:37:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:37:09.809 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:37:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:37:09.809 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:37:09.809 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:37:09.809 139328 ERROR oslo_service.periodic_task >2018-06-25 00:37:39.681 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:37:39.682 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:37:39.690 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:37:39.690 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:37:39.727 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:37:39.727 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:37:39.727 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:37:39.727 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:37:39.727 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:37:39.727 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:37:39.727 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:37:39.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:37:39.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:37:39.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:37:49.731 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:37:49.765 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:37:49.765 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:37:49.765 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:37:49.765 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:37:49.765 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:37:49.765 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:37:49.765 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:37:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:37:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:37:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:38:09.769 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:38:09.805 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:38:09.805 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:38:09.805 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:38:09.805 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:38:09.805 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:38:09.805 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:38:09.805 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:38:09.808 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:38:09.808 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:38:09.808 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:38:09.808 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:38:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:38:09.808 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:38:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:38:09.808 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:38:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:38:09.808 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:38:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:38:09.808 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:38:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:38:09.808 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:38:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:38:09.808 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:38:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:38:09.808 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:38:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:38:09.808 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:38:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:38:09.808 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:38:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:38:09.808 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:38:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:38:09.808 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:38:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:38:09.808 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:38:09.808 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:38:09.808 139328 ERROR oslo_service.periodic_task >2018-06-25 00:38:39.680 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:38:39.681 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:38:39.692 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:38:39.693 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:38:39.726 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:38:39.726 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:38:39.726 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:38:39.726 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:38:39.726 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:38:39.726 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:38:39.726 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:38:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:38:39.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:38:39.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:38:49.729 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:38:49.763 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:38:49.763 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:38:49.763 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:38:49.763 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:38:49.763 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:38:49.763 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:38:49.763 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:38:49.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:38:49.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:38:49.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:39:09.766 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:39:09.804 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:39:09.804 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:39:09.804 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:39:09.804 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:39:09.804 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:39:09.804 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:39:09.804 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:39:09.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:39:09.807 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:39:09.807 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:39:09.807 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:39:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:39:09.807 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:39:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:39:09.807 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:39:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:39:09.807 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:39:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:39:09.807 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:39:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:39:09.807 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:39:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:39:09.807 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:39:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:39:09.807 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:39:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:39:09.807 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:39:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:39:09.807 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:39:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:39:09.807 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:39:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:39:09.807 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:39:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:39:09.807 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:39:09.807 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:39:09.807 139328 ERROR oslo_service.periodic_task >2018-06-25 00:39:39.679 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:39:39.680 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:39:39.687 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:39:39.688 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:39:39.721 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:39:39.721 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:39:39.721 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:39:39.721 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:39:39.721 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:39:39.721 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:39:39.721 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:39:39.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:39:39.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:39:39.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:39:49.724 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:39:49.756 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:39:49.756 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:39:49.756 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:39:49.756 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:39:49.756 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:39:49.756 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:39:49.756 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:39:49.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:39:49.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.07 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:39:49.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:40:09.759 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:40:09.796 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:40:09.796 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:40:09.796 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:40:09.796 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:40:09.796 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:40:09.796 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:40:09.796 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:40:09.798 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:40:09.798 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:40:09.799 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:40:09.799 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:40:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:40:09.799 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:40:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:40:09.799 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:40:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:40:09.799 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:40:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:40:09.799 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:40:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:40:09.799 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:40:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:40:09.799 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:40:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:40:09.799 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:40:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:40:09.799 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:40:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:40:09.799 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:40:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:40:09.799 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:40:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:40:09.799 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:40:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:40:09.799 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:40:09.799 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:40:09.799 139328 ERROR oslo_service.periodic_task >2018-06-25 00:40:39.680 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:40:39.681 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:40:39.689 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:40:39.689 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:40:39.725 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:40:39.725 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:40:39.725 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:40:39.725 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:40:39.725 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:40:39.725 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:40:39.725 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:40:39.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:40:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:40:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:40:49.729 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:40:49.761 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:40:49.761 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:40:49.761 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:40:49.761 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:40:49.761 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:40:49.761 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:40:49.761 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:40:49.764 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:40:49.764 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:40:49.764 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:41:09.765 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:41:09.802 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:41:09.802 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:41:09.802 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:41:09.802 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:41:09.802 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:41:09.802 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:41:09.802 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:41:09.804 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:41:09.805 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:41:09.805 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:41:09.805 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:41:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:41:09.805 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:41:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:41:09.805 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:41:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:41:09.805 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:41:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:41:09.805 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:41:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:41:09.805 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:41:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:41:09.805 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:41:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:41:09.805 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:41:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:41:09.805 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:41:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:41:09.805 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:41:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:41:09.805 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:41:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:41:09.805 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:41:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:41:09.805 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:41:09.805 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:41:09.805 139328 ERROR oslo_service.periodic_task >2018-06-25 00:41:39.677 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:41:39.678 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:41:39.685 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:41:39.686 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:41:39.718 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:41:39.718 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:41:39.718 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:41:39.718 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:41:39.718 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:41:39.718 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:41:39.718 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:41:39.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:41:39.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:41:39.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:41:49.722 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:41:49.755 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:41:49.755 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:41:49.755 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:41:49.755 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:41:49.755 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:41:49.755 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:41:49.755 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:41:49.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:41:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:41:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:42:09.758 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:42:09.788 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:42:09.788 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:42:09.788 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:42:09.788 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:42:09.788 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:42:09.788 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:42:09.788 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:42:09.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:42:09.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.105 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:42:09.791 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:42:09.791 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:42:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:42:09.791 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:42:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:42:09.791 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:42:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:42:09.791 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:42:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:42:09.791 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:42:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:42:09.791 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:42:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:42:09.791 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:42:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:42:09.791 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:42:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:42:09.791 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:42:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:42:09.791 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:42:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:42:09.791 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:42:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:42:09.791 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:42:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:42:09.791 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:42:09.791 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:42:09.791 139328 ERROR oslo_service.periodic_task >2018-06-25 00:42:39.672 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:42:39.673 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:42:39.679 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:42:39.679 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:42:39.712 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:42:39.712 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:42:39.712 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:42:39.712 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:42:39.712 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:42:39.712 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:42:39.712 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:42:39.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:42:39.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:42:39.714 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:42:49.715 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:42:49.748 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:42:49.748 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:42:49.748 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:42:49.748 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:42:49.748 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:42:49.748 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:42:49.748 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:42:49.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:42:49.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.07 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:42:49.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:43:09.751 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:43:09.786 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:43:09.786 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:43:09.786 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:43:09.786 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:43:09.786 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:43:09.786 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:43:09.786 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:43:09.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:43:09.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:43:09.789 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:43:09.789 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:43:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:43:09.789 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:43:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:43:09.789 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:43:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:43:09.789 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:43:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:43:09.789 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:43:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:43:09.789 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:43:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:43:09.789 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:43:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:43:09.789 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:43:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:43:09.789 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:43:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:43:09.789 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:43:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:43:09.789 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:43:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:43:09.789 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:43:09.789 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:43:09.789 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:43:09.789 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:43:09.789 139328 ERROR oslo_service.periodic_task >2018-06-25 00:43:39.670 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:43:39.671 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:43:39.680 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:43:39.680 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:43:39.713 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:43:39.713 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:43:39.713 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:43:39.713 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:43:39.713 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:43:39.713 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:43:39.713 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:43:39.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:43:39.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:43:39.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:43:49.717 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:43:49.749 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:43:49.749 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:43:49.749 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:43:49.749 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:43:49.749 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:43:49.749 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:43:49.749 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:43:49.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:43:49.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:43:49.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:44:09.752 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:44:09.785 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:44:09.785 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:44:09.785 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:44:09.785 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:44:09.785 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:44:09.785 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:44:09.785 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:44:09.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:44:09.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.107 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:44:09.788 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:44:09.788 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:44:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:44:09.788 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:44:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:44:09.788 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:44:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:44:09.788 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:44:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:44:09.788 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:44:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:44:09.788 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:44:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:44:09.788 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:44:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:44:09.788 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:44:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:44:09.788 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:44:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:44:09.788 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:44:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:44:09.788 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:44:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:44:09.788 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:44:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:44:09.788 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:44:09.788 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:44:09.788 139328 ERROR oslo_service.periodic_task >2018-06-25 00:44:39.669 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:44:39.670 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:44:39.676 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:44:39.677 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:44:39.709 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:44:39.709 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:44:39.709 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:44:39.709 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:44:39.709 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:44:39.709 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:44:39.709 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:44:39.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:44:39.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:44:39.712 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:44:49.713 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:44:49.748 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:44:49.748 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:44:49.748 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:44:49.748 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:44:49.748 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:44:49.748 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:44:49.748 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:44:49.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:44:49.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:44:49.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:45:09.751 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:45:09.785 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:45:09.785 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:45:09.785 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:45:09.785 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:45:09.785 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:45:09.785 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:45:09.785 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:45:09.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:45:09.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:45:09.787 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:45:09.787 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:45:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:45:09.787 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:45:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:45:09.787 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:45:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:45:09.787 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:45:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:45:09.787 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:45:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:45:09.787 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:45:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:45:09.787 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:45:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:45:09.787 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:45:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:45:09.787 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:45:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:45:09.787 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:45:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:45:09.787 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:45:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:45:09.787 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:45:09.787 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:45:09.787 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:45:09.787 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:45:09.787 139328 ERROR oslo_service.periodic_task >2018-06-25 00:45:39.669 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:45:39.670 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:45:39.677 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:45:39.677 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:45:39.713 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:45:39.713 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:45:39.713 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:45:39.713 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:45:39.713 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:45:39.713 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:45:39.713 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:45:39.715 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:45:39.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:45:39.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:45:49.716 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:45:49.772 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:45:49.772 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:45:49.772 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:45:49.772 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:45:49.772 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:45:49.772 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:45:49.772 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:45:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:45:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.096 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:45:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:46:09.775 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:46:09.808 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:46:09.808 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:46:09.808 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:46:09.808 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:46:09.808 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:46:09.808 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:46:09.808 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:46:09.810 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:46:09.811 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.133 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:46:09.811 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:46:09.811 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:46:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:46:09.811 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:46:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:46:09.811 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:46:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:46:09.811 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:46:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:46:09.811 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:46:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:46:09.811 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:46:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:46:09.811 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:46:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:46:09.811 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:46:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:46:09.811 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:46:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:46:09.811 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:46:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:46:09.811 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:46:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:46:09.811 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:46:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:46:09.811 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:46:09.811 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:46:09.811 139328 ERROR oslo_service.periodic_task >2018-06-25 00:46:39.672 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:46:39.674 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:46:39.682 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:46:39.682 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:46:39.717 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:46:39.717 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:46:39.717 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:46:39.717 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:46:39.717 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:46:39.717 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:46:39.717 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:46:39.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:46:39.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:46:39.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:46:49.720 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:46:49.754 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:46:49.754 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:46:49.754 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:46:49.754 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:46:49.754 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:46:49.754 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:46:49.754 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:46:49.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:46:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:46:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:47:09.758 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:47:09.796 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:47:09.796 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:47:09.796 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:47:09.796 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:47:09.796 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:47:09.796 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:47:09.796 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:47:09.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:47:09.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:47:09.800 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:47:09.800 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:47:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:47:09.800 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:47:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:47:09.800 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:47:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:47:09.800 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:47:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:47:09.800 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:47:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:47:09.800 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:47:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:47:09.800 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:47:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:47:09.800 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:47:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:47:09.800 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:47:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:47:09.800 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:47:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:47:09.800 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:47:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:47:09.800 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:47:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:47:09.800 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:47:09.800 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:47:09.800 139328 ERROR oslo_service.periodic_task >2018-06-25 00:47:39.672 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:47:39.672 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:47:39.680 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:47:39.681 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:47:39.713 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:47:39.713 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:47:39.713 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:47:39.713 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:47:39.713 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:47:39.713 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:47:39.713 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:47:39.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:47:39.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:47:39.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:47:49.717 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:47:49.767 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:47:49.767 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:47:49.767 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:47:49.767 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:47:49.767 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:47:49.767 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:47:49.767 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:47:49.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:47:49.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.089 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:47:49.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:48:09.771 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:48:09.808 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:48:09.808 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:48:09.808 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:48:09.808 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:48:09.808 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:48:09.808 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:48:09.808 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:48:09.810 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:48:09.811 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.13 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:48:09.811 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:48:09.811 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:48:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:48:09.811 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:48:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:48:09.811 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:48:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:48:09.811 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:48:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:48:09.811 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:48:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:48:09.811 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:48:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:48:09.811 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:48:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:48:09.811 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:48:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:48:09.811 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:48:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:48:09.811 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:48:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:48:09.811 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:48:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:48:09.811 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:48:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:48:09.811 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:48:09.811 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:48:09.811 139328 ERROR oslo_service.periodic_task >2018-06-25 00:48:39.672 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:48:39.674 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:48:39.680 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:48:39.680 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:48:39.713 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:48:39.713 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:48:39.713 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:48:39.713 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:48:39.713 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:48:39.713 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:48:39.713 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:48:39.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:48:39.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:48:39.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:48:49.717 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:48:49.750 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:48:49.750 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:48:49.750 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:48:49.750 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:48:49.750 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:48:49.750 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:48:49.750 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:48:49.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:48:49.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:48:49.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:49:09.753 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:49:09.785 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:49:09.785 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:49:09.785 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:49:09.785 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:49:09.785 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:49:09.785 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:49:09.785 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:49:09.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:49:09.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.106 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:49:09.788 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:49:09.788 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:49:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:49:09.788 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:49:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:49:09.788 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:49:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:49:09.788 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:49:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:49:09.788 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:49:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:49:09.788 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:49:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:49:09.788 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:49:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:49:09.788 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:49:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:49:09.788 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:49:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:49:09.788 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:49:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:49:09.788 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:49:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:49:09.788 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:49:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:49:09.788 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:49:09.788 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:49:09.788 139328 ERROR oslo_service.periodic_task >2018-06-25 00:49:39.669 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:49:39.670 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:49:39.703 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:49:39.703 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:49:39.703 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:49:39.703 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:49:39.703 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:49:39.703 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:49:39.703 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:49:39.705 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:49:39.705 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:49:39.706 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:49:49.706 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:49:49.749 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:49:49.749 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:49:49.749 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:49:49.749 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:49:49.749 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:49:49.749 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:49:49.749 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:49:49.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:49:49.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.081 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:49:49.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:50:09.752 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:50:09.785 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:50:09.785 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:50:09.785 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:50:09.785 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:50:09.785 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:50:09.785 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:50:09.785 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:50:09.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:50:09.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:50:09.788 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:50:09.788 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:50:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:50:09.788 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:50:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:50:09.788 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:50:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:50:09.788 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:50:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:50:09.788 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:50:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:50:09.788 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:50:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:50:09.788 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:50:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:50:09.788 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:50:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:50:09.788 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:50:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:50:09.788 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:50:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:50:09.788 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:50:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:50:09.788 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:50:09.788 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:50:09.788 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:50:09.788 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:50:09.788 139328 ERROR oslo_service.periodic_task >2018-06-25 00:50:39.669 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:50:39.670 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:50:39.681 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:50:39.682 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:50:39.717 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:50:39.717 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:50:39.717 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:50:39.717 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:50:39.717 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:50:39.717 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:50:39.717 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:50:39.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:50:39.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:50:39.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:50:49.720 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:50:49.752 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:50:49.752 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:50:49.752 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:50:49.752 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:50:49.752 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:50:49.752 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:50:49.752 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:50:49.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:50:49.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:50:49.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:51:09.755 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:51:09.788 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:51:09.788 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:51:09.788 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:51:09.788 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:51:09.788 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:51:09.788 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:51:09.788 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:51:09.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:51:09.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:51:09.791 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:51:09.791 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:51:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:51:09.791 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:51:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:51:09.791 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:51:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:51:09.791 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:51:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:51:09.791 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:51:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:51:09.791 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:51:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:51:09.791 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:51:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:51:09.791 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:51:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:51:09.791 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:51:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:51:09.791 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:51:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:51:09.791 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:51:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:51:09.791 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:51:09.791 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:51:09.791 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:51:09.791 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:51:09.791 139328 ERROR oslo_service.periodic_task >2018-06-25 00:51:39.672 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:51:39.673 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:51:39.680 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:51:39.681 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:51:39.715 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:51:39.715 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:51:39.715 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:51:39.715 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:51:39.715 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:51:39.715 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:51:39.715 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:51:39.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:51:39.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:51:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:51:49.719 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:51:49.752 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:51:49.752 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:51:49.752 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:51:49.752 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:51:49.752 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:51:49.752 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:51:49.752 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:51:49.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:51:49.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:51:49.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:52:09.755 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:52:09.791 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:52:09.791 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:52:09.791 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:52:09.791 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:52:09.791 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:52:09.791 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:52:09.791 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:52:09.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:52:09.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:52:09.794 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:52:09.794 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:52:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:52:09.794 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:52:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:52:09.794 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:52:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:52:09.794 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:52:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:52:09.794 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:52:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:52:09.794 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:52:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:52:09.794 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:52:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:52:09.794 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:52:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:52:09.794 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:52:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:52:09.794 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:52:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:52:09.794 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:52:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:52:09.794 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:52:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:52:09.794 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:52:09.794 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:52:09.794 139328 ERROR oslo_service.periodic_task >2018-06-25 00:52:39.675 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:52:39.676 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:52:39.686 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:52:39.687 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:52:39.723 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:52:39.723 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:52:39.723 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:52:39.723 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:52:39.723 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:52:39.723 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:52:39.723 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:52:39.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:52:39.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:52:39.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:52:49.727 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:52:49.760 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:52:49.760 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:52:49.760 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:52:49.760 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:52:49.760 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:52:49.760 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:52:49.760 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:52:49.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:52:49.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:52:49.764 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:53:09.765 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:53:09.798 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:53:09.798 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:53:09.798 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:53:09.798 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:53:09.798 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:53:09.798 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:53:09.798 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:53:09.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:53:09.801 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:53:09.801 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:53:09.801 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:53:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:53:09.801 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:53:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:53:09.801 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:53:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:53:09.801 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:53:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:53:09.801 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:53:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:53:09.801 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:53:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:53:09.801 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:53:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:53:09.801 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:53:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:53:09.801 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:53:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:53:09.801 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:53:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:53:09.801 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:53:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:53:09.801 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:53:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:53:09.801 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:53:09.801 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:53:09.801 139328 ERROR oslo_service.periodic_task >2018-06-25 00:53:39.673 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:53:39.674 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:53:39.681 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:53:39.681 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:53:39.714 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:53:39.714 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:53:39.714 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:53:39.714 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:53:39.714 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:53:39.714 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:53:39.714 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:53:39.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:53:39.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:53:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:53:49.718 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:53:49.750 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:53:49.750 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:53:49.750 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:53:49.750 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:53:49.750 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:53:49.750 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:53:49.750 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:53:49.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:53:49.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:53:49.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:54:09.753 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:54:09.788 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:54:09.788 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:54:09.788 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:54:09.788 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:54:09.788 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:54:09.788 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:54:09.788 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:54:09.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:54:09.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:54:09.790 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:54:09.790 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:54:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:54:09.790 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:54:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:54:09.790 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:54:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:54:09.790 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:54:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:54:09.790 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:54:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:54:09.790 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:54:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:54:09.790 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:54:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:54:09.790 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:54:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:54:09.790 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:54:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:54:09.790 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:54:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:54:09.790 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:54:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:54:09.790 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:54:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:54:09.790 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:54:09.790 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:54:09.790 139328 ERROR oslo_service.periodic_task >2018-06-25 00:54:39.672 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:54:39.673 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:54:39.680 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:54:39.680 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:54:39.714 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:54:39.714 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:54:39.714 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:54:39.714 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:54:39.714 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:54:39.714 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:54:39.714 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:54:39.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:54:39.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:54:39.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:54:49.717 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:54:49.749 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:54:49.749 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:54:49.749 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:54:49.749 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:54:49.749 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:54:49.749 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:54:49.749 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:54:49.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:54:49.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:54:49.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:55:09.753 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:55:09.787 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:55:09.787 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:55:09.787 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:55:09.787 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:55:09.787 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:55:09.787 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:55:09.787 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:55:09.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:55:09.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:55:09.790 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:55:09.790 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:55:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:55:09.790 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:55:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:55:09.790 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:55:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:55:09.790 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:55:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:55:09.790 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:55:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:55:09.790 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:55:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:55:09.790 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:55:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:55:09.790 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:55:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:55:09.790 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:55:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:55:09.790 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:55:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:55:09.790 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:55:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:55:09.790 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:55:09.790 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:55:09.790 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:55:09.790 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:55:09.790 139328 ERROR oslo_service.periodic_task >2018-06-25 00:55:39.671 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:55:39.672 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:55:39.680 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:55:39.680 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:55:39.714 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:55:39.714 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:55:39.714 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:55:39.714 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:55:39.714 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:55:39.714 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:55:39.714 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:55:39.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:55:39.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:55:39.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:55:49.717 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:55:49.751 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:55:49.751 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:55:49.751 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:55:49.751 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:55:49.751 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:55:49.751 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:55:49.751 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:55:49.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:55:49.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:55:49.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:56:09.755 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:56:09.790 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:56:09.790 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:56:09.790 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:56:09.790 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:56:09.790 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:56:09.790 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:56:09.790 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:56:09.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:56:09.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:56:09.793 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:56:09.793 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:56:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:56:09.793 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:56:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:56:09.793 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:56:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:56:09.793 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:56:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:56:09.793 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:56:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:56:09.793 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:56:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:56:09.793 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:56:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:56:09.793 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:56:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:56:09.793 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:56:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:56:09.793 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:56:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:56:09.793 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:56:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:56:09.793 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:56:09.793 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:56:09.793 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:56:09.793 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:56:09.793 139328 ERROR oslo_service.periodic_task >2018-06-25 00:56:39.674 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:56:39.676 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:56:39.682 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:56:39.683 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:56:39.717 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:56:39.717 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:56:39.717 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:56:39.717 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:56:39.717 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:56:39.717 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:56:39.717 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:56:39.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:56:39.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:56:39.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:56:49.720 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:56:49.757 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:56:49.757 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:56:49.757 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:56:49.757 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:56:49.757 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:56:49.757 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:56:49.757 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:56:49.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:56:49.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:56:49.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:57:09.760 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:57:09.795 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:57:09.795 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:57:09.795 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:57:09.795 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:57:09.795 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:57:09.795 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:57:09.795 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:57:09.797 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:57:09.797 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:57:09.797 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:57:09.797 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:57:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:57:09.797 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:57:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:57:09.797 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:57:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:57:09.797 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:57:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:57:09.797 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:57:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:57:09.797 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:57:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:57:09.797 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:57:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:57:09.797 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:57:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:57:09.797 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:57:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:57:09.797 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:57:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:57:09.797 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:57:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:57:09.797 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:57:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:57:09.797 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:57:09.797 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:57:09.797 139328 ERROR oslo_service.periodic_task >2018-06-25 00:57:39.679 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:57:39.680 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:57:39.687 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:57:39.687 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:57:39.721 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:57:39.721 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:57:39.721 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:57:39.721 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:57:39.721 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:57:39.721 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:57:39.721 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:57:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:57:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:57:39.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:57:49.725 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:57:49.760 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:57:49.760 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:57:49.760 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:57:49.760 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:57:49.760 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:57:49.760 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:57:49.760 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:57:49.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:57:49.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:57:49.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:58:09.764 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:58:09.799 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:58:09.799 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:58:09.799 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:58:09.799 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:58:09.799 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:58:09.799 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:58:09.799 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:58:09.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:58:09.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:58:09.803 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:58:09.803 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:58:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:58:09.803 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:58:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:58:09.803 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:58:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:58:09.803 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:58:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:58:09.803 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:58:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:58:09.803 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:58:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:58:09.803 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:58:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:58:09.803 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:58:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:58:09.803 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:58:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:58:09.803 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:58:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:58:09.803 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:58:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:58:09.803 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:58:09.803 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:58:09.803 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:58:09.803 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:58:09.803 139328 ERROR oslo_service.periodic_task >2018-06-25 00:58:39.674 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:58:39.675 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:58:39.682 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:58:39.683 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:58:39.718 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:58:39.718 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:58:39.718 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:58:39.718 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:58:39.718 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:58:39.718 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:58:39.718 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:58:39.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:58:39.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:58:39.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:58:49.722 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:58:49.754 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:58:49.754 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:58:49.754 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:58:49.754 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:58:49.754 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:58:49.754 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:58:49.754 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:58:49.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:58:49.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:58:49.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:59:09.759 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:59:09.792 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:59:09.792 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:59:09.792 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:59:09.792 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:59:09.792 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:59:09.792 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:59:09.792 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:59:09.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:59:09.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:59:09.795 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:59:09.795 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 00:59:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 00:59:09.795 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 00:59:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 00:59:09.795 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 00:59:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 00:59:09.795 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 00:59:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 00:59:09.795 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 00:59:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 00:59:09.795 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 00:59:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 00:59:09.795 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 00:59:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 00:59:09.795 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 00:59:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 00:59:09.795 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 00:59:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 00:59:09.795 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 00:59:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 00:59:09.795 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 00:59:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 00:59:09.795 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 00:59:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 00:59:09.795 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 00:59:09.795 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 00:59:09.795 139328 ERROR oslo_service.periodic_task >2018-06-25 00:59:39.677 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:59:39.678 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 00:59:39.685 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 00:59:39.685 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:59:39.721 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:59:39.721 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:59:39.721 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:59:39.721 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:59:39.721 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:59:39.721 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:59:39.721 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:59:39.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:59:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:59:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 00:59:49.725 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 00:59:49.757 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 00:59:49.757 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 00:59:49.757 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 00:59:49.757 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 00:59:49.757 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 00:59:49.757 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 00:59:49.757 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 00:59:49.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 00:59:49.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 00:59:49.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:00:09.760 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:00:09.797 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:00:09.797 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:00:09.797 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:00:09.797 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:00:09.797 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:00:09.797 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:00:09.797 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:00:09.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:00:09.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:00:09.800 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:00:09.800 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:00:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:00:09.800 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:00:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:00:09.800 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:00:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:00:09.800 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:00:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:00:09.800 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:00:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:00:09.800 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:00:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:00:09.800 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:00:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:00:09.800 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:00:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:00:09.800 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:00:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:00:09.800 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:00:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:00:09.800 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:00:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:00:09.800 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:00:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:00:09.800 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:00:09.800 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:00:09.800 139328 ERROR oslo_service.periodic_task >2018-06-25 01:00:39.682 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:00:39.683 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:00:39.690 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:00:39.691 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:00:39.725 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:00:39.725 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:00:39.725 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:00:39.725 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:00:39.725 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:00:39.725 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:00:39.725 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:00:39.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:00:39.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:00:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:00:49.728 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:00:49.759 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:00:49.759 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:00:49.759 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:00:49.759 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:00:49.759 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:00:49.759 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:00:49.759 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:00:49.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:00:49.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.07 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:00:49.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:01:09.762 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:01:09.799 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:01:09.799 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:01:09.799 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:01:09.799 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:01:09.799 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:01:09.799 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:01:09.799 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:01:09.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:01:09.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:01:09.802 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:01:09.802 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:01:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:01:09.802 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:01:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:01:09.802 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:01:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:01:09.802 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:01:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:01:09.802 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:01:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:01:09.802 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:01:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:01:09.802 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:01:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:01:09.802 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:01:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:01:09.802 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:01:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:01:09.802 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:01:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:01:09.802 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:01:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:01:09.802 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:01:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:01:09.802 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:01:09.802 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:01:09.802 139328 ERROR oslo_service.periodic_task >2018-06-25 01:01:39.684 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:01:39.685 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:01:39.692 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:01:39.693 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:01:39.726 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:01:39.726 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:01:39.726 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:01:39.726 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:01:39.726 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:01:39.726 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:01:39.726 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:01:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:01:39.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:01:39.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:01:49.730 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:01:49.763 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:01:49.763 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:01:49.763 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:01:49.763 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:01:49.763 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:01:49.763 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:01:49.763 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:01:49.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:01:49.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:01:49.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:02:09.766 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:02:09.808 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:02:09.808 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:02:09.808 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:02:09.808 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:02:09.808 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:02:09.808 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:02:09.808 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:02:09.810 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:02:09.810 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:02:09.811 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:02:09.811 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:02:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:02:09.811 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:02:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:02:09.811 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:02:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:02:09.811 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:02:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:02:09.811 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:02:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:02:09.811 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:02:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:02:09.811 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:02:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:02:09.811 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:02:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:02:09.811 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:02:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:02:09.811 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:02:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:02:09.811 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:02:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:02:09.811 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:02:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:02:09.811 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:02:09.811 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:02:09.811 139328 ERROR oslo_service.periodic_task >2018-06-25 01:02:39.682 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:02:39.683 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:02:39.691 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:02:39.691 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:02:39.727 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:02:39.727 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:02:39.727 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:02:39.727 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:02:39.727 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:02:39.727 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:02:39.727 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:02:39.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:02:39.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:02:39.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:02:49.731 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:02:49.763 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:02:49.763 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:02:49.763 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:02:49.763 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:02:49.763 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:02:49.763 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:02:49.763 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:02:49.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:02:49.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:02:49.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:03:09.766 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:03:09.798 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:03:09.798 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:03:09.798 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:03:09.798 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:03:09.798 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:03:09.798 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:03:09.798 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:03:09.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:03:09.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:03:09.801 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:03:09.801 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:03:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:03:09.801 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:03:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:03:09.801 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:03:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:03:09.801 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:03:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:03:09.801 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:03:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:03:09.801 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:03:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:03:09.801 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:03:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:03:09.801 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:03:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:03:09.801 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:03:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:03:09.801 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:03:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:03:09.801 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:03:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:03:09.801 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:03:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:03:09.801 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:03:09.801 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:03:09.801 139328 ERROR oslo_service.periodic_task >2018-06-25 01:03:39.682 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:03:39.683 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:03:39.692 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:03:39.693 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:03:39.731 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:03:39.731 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:03:39.731 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:03:39.731 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:03:39.731 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:03:39.731 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:03:39.731 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:03:39.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:03:39.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:03:39.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:03:49.734 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:03:49.778 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:03:49.778 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:03:49.778 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:03:49.778 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:03:49.778 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:03:49.778 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:03:49.778 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:03:49.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:03:49.783 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.089 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:03:49.783 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:04:09.783 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:04:09.815 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:04:09.815 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:04:09.815 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:04:09.815 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:04:09.815 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:04:09.815 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:04:09.815 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:04:09.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:04:09.818 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.124 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:04:09.818 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:04:09.818 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:04:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:04:09.818 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:04:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:04:09.818 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:04:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:04:09.818 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:04:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:04:09.818 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:04:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:04:09.818 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:04:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:04:09.818 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:04:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:04:09.818 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:04:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:04:09.818 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:04:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:04:09.818 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:04:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:04:09.818 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:04:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:04:09.818 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:04:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:04:09.818 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:04:09.818 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:04:09.818 139328 ERROR oslo_service.periodic_task >2018-06-25 01:04:39.680 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:04:39.680 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:04:39.687 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:04:39.688 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:04:39.721 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:04:39.721 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:04:39.721 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:04:39.721 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:04:39.721 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:04:39.721 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:04:39.721 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:04:39.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:04:39.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:04:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:04:49.724 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:04:49.760 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:04:49.760 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:04:49.760 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:04:49.760 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:04:49.760 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:04:49.760 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:04:49.760 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:04:49.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:04:49.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:04:49.764 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:05:09.764 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:05:09.796 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:05:09.796 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:05:09.796 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:05:09.796 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:05:09.796 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:05:09.796 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:05:09.796 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:05:09.798 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:05:09.798 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:05:09.798 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:05:09.798 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:05:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:05:09.798 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:05:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:05:09.798 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:05:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:05:09.798 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:05:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:05:09.798 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:05:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:05:09.798 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:05:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:05:09.798 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:05:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:05:09.798 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:05:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:05:09.798 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:05:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:05:09.798 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:05:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:05:09.798 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:05:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:05:09.798 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:05:09.798 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:05:09.798 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:05:09.798 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:05:09.798 139328 ERROR oslo_service.periodic_task >2018-06-25 01:05:39.680 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:05:39.680 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:05:39.687 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:05:39.688 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:05:39.723 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:05:39.723 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:05:39.723 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:05:39.723 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:05:39.723 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:05:39.723 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:05:39.723 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:05:39.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:05:39.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:05:39.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:05:49.726 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:05:49.760 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:05:49.760 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:05:49.760 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:05:49.760 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:05:49.760 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:05:49.760 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:05:49.760 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:05:49.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:05:49.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:05:49.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:06:09.763 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:06:09.798 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:06:09.798 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:06:09.798 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:06:09.798 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:06:09.798 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:06:09.798 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:06:09.798 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:06:09.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:06:09.801 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:06:09.801 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:06:09.801 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:06:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:06:09.801 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:06:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:06:09.801 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:06:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:06:09.801 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:06:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:06:09.801 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:06:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:06:09.801 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:06:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:06:09.801 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:06:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:06:09.801 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:06:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:06:09.801 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:06:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:06:09.801 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:06:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:06:09.801 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:06:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:06:09.801 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:06:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:06:09.801 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:06:09.801 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:06:09.801 139328 ERROR oslo_service.periodic_task >2018-06-25 01:06:39.683 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:06:39.685 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:06:39.693 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:06:39.694 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:06:39.735 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:06:39.735 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:06:39.735 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:06:39.735 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:06:39.735 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:06:39.735 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:06:39.735 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:06:39.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:06:39.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.043 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:06:39.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:06:49.738 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:06:49.771 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:06:49.771 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:06:49.771 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:06:49.771 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:06:49.771 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:06:49.771 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:06:49.771 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:06:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:06:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:06:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:07:09.774 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:07:09.809 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:07:09.809 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:07:09.809 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:07:09.809 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:07:09.809 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:07:09.809 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:07:09.809 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:07:09.812 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:07:09.812 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:07:09.813 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:07:09.813 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:07:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:07:09.813 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:07:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:07:09.813 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:07:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:07:09.813 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:07:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:07:09.813 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:07:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:07:09.813 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:07:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:07:09.813 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:07:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:07:09.813 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:07:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:07:09.813 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:07:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:07:09.813 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:07:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:07:09.813 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:07:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:07:09.813 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:07:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:07:09.813 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:07:09.813 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:07:09.813 139328 ERROR oslo_service.periodic_task >2018-06-25 01:07:39.684 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:07:39.685 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:07:39.693 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:07:39.694 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:07:39.729 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:07:39.729 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:07:39.729 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:07:39.729 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:07:39.729 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:07:39.729 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:07:39.729 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:07:39.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:07:39.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:07:39.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:07:49.733 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:07:49.768 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:07:49.768 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:07:49.768 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:07:49.768 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:07:49.768 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:07:49.768 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:07:49.768 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:07:49.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:07:49.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:07:49.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:08:09.772 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:08:09.805 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:08:09.805 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:08:09.805 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:08:09.805 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:08:09.805 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:08:09.805 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:08:09.805 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:08:09.807 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:08:09.807 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:08:09.808 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:08:09.808 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:08:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:08:09.808 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:08:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:08:09.808 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:08:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:08:09.808 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:08:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:08:09.808 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:08:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:08:09.808 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:08:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:08:09.808 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:08:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:08:09.808 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:08:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:08:09.808 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:08:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:08:09.808 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:08:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:08:09.808 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:08:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:08:09.808 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:08:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:08:09.808 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:08:09.808 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:08:09.808 139328 ERROR oslo_service.periodic_task >2018-06-25 01:08:39.689 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:08:39.690 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:08:39.698 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:08:39.699 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:08:39.732 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:08:39.732 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:08:39.732 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:08:39.732 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:08:39.732 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:08:39.732 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:08:39.732 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:08:39.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:08:39.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:08:39.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:08:49.735 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:08:49.768 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:08:49.768 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:08:49.768 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:08:49.768 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:08:49.768 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:08:49.768 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:08:49.768 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:08:49.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:08:49.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:08:49.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:09:09.771 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:09:09.810 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:09:09.810 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:09:09.810 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:09:09.810 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:09:09.810 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:09:09.810 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:09:09.810 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:09:09.813 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:09:09.813 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:09:09.814 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:09:09.814 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:09:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:09:09.814 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:09:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:09:09.814 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:09:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:09:09.814 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:09:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:09:09.814 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:09:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:09:09.814 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:09:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:09:09.814 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:09:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:09:09.814 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:09:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:09:09.814 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:09:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:09:09.814 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:09:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:09:09.814 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:09:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:09:09.814 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:09:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:09:09.814 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:09:09.814 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:09:09.814 139328 ERROR oslo_service.periodic_task >2018-06-25 01:09:39.686 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:09:39.687 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:09:39.698 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:09:39.699 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:09:39.737 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:09:39.737 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:09:39.737 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:09:39.737 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:09:39.737 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:09:39.737 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:09:39.737 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:09:39.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:09:39.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:09:39.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:09:49.740 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:09:49.773 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:09:49.773 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:09:49.773 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:09:49.773 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:09:49.773 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:09:49.773 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:09:49.773 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:09:49.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:09:49.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:09:49.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:10:09.777 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:10:09.814 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:10:09.814 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:10:09.814 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:10:09.814 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:10:09.814 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:10:09.814 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:10:09.814 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:10:09.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:10:09.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:10:09.817 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:10:09.817 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:10:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:10:09.817 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:10:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:10:09.817 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:10:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:10:09.817 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:10:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:10:09.817 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:10:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:10:09.817 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:10:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:10:09.817 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:10:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:10:09.817 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:10:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:10:09.817 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:10:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:10:09.817 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:10:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:10:09.817 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:10:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:10:09.817 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:10:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:10:09.817 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:10:09.817 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:10:09.817 139328 ERROR oslo_service.periodic_task >2018-06-25 01:10:39.688 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:10:39.689 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:10:39.697 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:10:39.698 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:10:39.732 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:10:39.732 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:10:39.732 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:10:39.732 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:10:39.732 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:10:39.732 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:10:39.732 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:10:39.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:10:39.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:10:39.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:10:49.735 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:10:49.768 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:10:49.768 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:10:49.768 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:10:49.768 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:10:49.768 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:10:49.768 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:10:49.768 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:10:49.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:10:49.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:10:49.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:11:09.771 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:11:09.823 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:11:09.823 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:11:09.823 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:11:09.823 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:11:09.823 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:11:09.823 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:11:09.823 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:11:09.825 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:11:09.826 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.128 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:11:09.826 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:11:09.826 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:11:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:11:09.826 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:11:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:11:09.826 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:11:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:11:09.826 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:11:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:11:09.826 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:11:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:11:09.826 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:11:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:11:09.826 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:11:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:11:09.826 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:11:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:11:09.826 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:11:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:11:09.826 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:11:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:11:09.826 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:11:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:11:09.826 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:11:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:11:09.826 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:11:09.826 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:11:09.826 139328 ERROR oslo_service.periodic_task >2018-06-25 01:11:39.688 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:11:39.689 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:11:39.698 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:11:39.699 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:11:39.731 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:11:39.731 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:11:39.731 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:11:39.731 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:11:39.731 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:11:39.731 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:11:39.731 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:11:39.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:11:39.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:11:39.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:11:49.735 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:11:49.771 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:11:49.771 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:11:49.771 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:11:49.771 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:11:49.771 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:11:49.771 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:11:49.771 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:11:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:11:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:11:49.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:12:09.775 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:12:09.808 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:12:09.808 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:12:09.808 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:12:09.808 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:12:09.808 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:12:09.808 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:12:09.808 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:12:09.810 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:12:09.810 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:12:09.810 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:12:09.810 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:12:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:12:09.810 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:12:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:12:09.810 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:12:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:12:09.810 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:12:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:12:09.810 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:12:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:12:09.810 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:12:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:12:09.810 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:12:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:12:09.810 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:12:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:12:09.810 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:12:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:12:09.810 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:12:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:12:09.810 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:12:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:12:09.810 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:12:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:12:09.810 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:12:09.810 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:12:09.810 139328 ERROR oslo_service.periodic_task >2018-06-25 01:12:39.692 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:12:39.693 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:12:39.700 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:12:39.700 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:12:39.735 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:12:39.735 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:12:39.735 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:12:39.735 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:12:39.735 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:12:39.735 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:12:39.735 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:12:39.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:12:39.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:12:39.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:12:49.738 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:12:49.772 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:12:49.772 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:12:49.772 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:12:49.772 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:12:49.772 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:12:49.772 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:12:49.772 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:12:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:12:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:12:49.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:13:09.775 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:13:09.809 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:13:09.809 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:13:09.809 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:13:09.809 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:13:09.809 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:13:09.809 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:13:09.809 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:13:09.812 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:13:09.812 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:13:09.813 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:13:09.813 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:13:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:13:09.813 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:13:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:13:09.813 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:13:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:13:09.813 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:13:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:13:09.813 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:13:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:13:09.813 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:13:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:13:09.813 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:13:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:13:09.813 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:13:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:13:09.813 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:13:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:13:09.813 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:13:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:13:09.813 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:13:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:13:09.813 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:13:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:13:09.813 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:13:09.813 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:13:09.813 139328 ERROR oslo_service.periodic_task >2018-06-25 01:13:39.694 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:13:39.695 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:13:39.703 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:13:39.704 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:13:39.737 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:13:39.737 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:13:39.737 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:13:39.737 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:13:39.737 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:13:39.737 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:13:39.737 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:13:39.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:13:39.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:13:39.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:13:49.740 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:13:49.774 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:13:49.774 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:13:49.774 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:13:49.774 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:13:49.774 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:13:49.774 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:13:49.774 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:13:49.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:13:49.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:13:49.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:14:09.779 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:14:09.812 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:14:09.812 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:14:09.812 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:14:09.812 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:14:09.812 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:14:09.812 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:14:09.812 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:14:09.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:14:09.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:14:09.815 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:14:09.815 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:14:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:14:09.815 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:14:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:14:09.815 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:14:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:14:09.815 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:14:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:14:09.815 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:14:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:14:09.815 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:14:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:14:09.815 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:14:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:14:09.815 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:14:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:14:09.815 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:14:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:14:09.815 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:14:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:14:09.815 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:14:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:14:09.815 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:14:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:14:09.815 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:14:09.815 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:14:09.815 139328 ERROR oslo_service.periodic_task >2018-06-25 01:14:39.697 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:14:39.698 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:14:39.705 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:14:39.706 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:14:39.746 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:14:39.746 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:14:39.746 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:14:39.746 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:14:39.746 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:14:39.746 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:14:39.746 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:14:39.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:14:39.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.043 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:14:39.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:14:49.749 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:14:49.783 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:14:49.783 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:14:49.783 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:14:49.783 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:14:49.783 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:14:49.783 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:14:49.783 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:14:49.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:14:49.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.08 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:14:49.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:15:09.786 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:15:09.820 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:15:09.820 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:15:09.820 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:15:09.820 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:15:09.820 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:15:09.820 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:15:09.820 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:15:09.822 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:15:09.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:15:09.823 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:15:09.823 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:15:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:15:09.823 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:15:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:15:09.823 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:15:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:15:09.823 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:15:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:15:09.823 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:15:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:15:09.823 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:15:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:15:09.823 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:15:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:15:09.823 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:15:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:15:09.823 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:15:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:15:09.823 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:15:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:15:09.823 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:15:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:15:09.823 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:15:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:15:09.823 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:15:09.823 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:15:09.823 139328 ERROR oslo_service.periodic_task >2018-06-25 01:15:39.695 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:15:39.696 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:15:39.703 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:15:39.703 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:15:39.740 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:15:39.740 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:15:39.740 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:15:39.740 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:15:39.740 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:15:39.740 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:15:39.740 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:15:39.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:15:39.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:15:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:15:49.743 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:15:49.777 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:15:49.777 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:15:49.777 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:15:49.777 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:15:49.777 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:15:49.777 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:15:49.777 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:15:49.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:15:49.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:15:49.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:16:09.780 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:16:09.814 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:16:09.814 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:16:09.814 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:16:09.814 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:16:09.814 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:16:09.814 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:16:09.814 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:16:09.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:16:09.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:16:09.816 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:16:09.816 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:16:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:16:09.816 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:16:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:16:09.816 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:16:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:16:09.816 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:16:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:16:09.816 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:16:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:16:09.816 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:16:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:16:09.816 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:16:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:16:09.816 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:16:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:16:09.816 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:16:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:16:09.816 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:16:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:16:09.816 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:16:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:16:09.816 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:16:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:16:09.816 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:16:09.816 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:16:09.816 139328 ERROR oslo_service.periodic_task >2018-06-25 01:16:39.697 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:16:39.699 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:16:39.707 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:16:39.707 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:16:39.742 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:16:39.742 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:16:39.742 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:16:39.742 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:16:39.742 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:16:39.742 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:16:39.742 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:16:39.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:16:39.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:16:39.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:16:49.746 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:16:49.782 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:16:49.782 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:16:49.782 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:16:49.782 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:16:49.782 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:16:49.782 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:16:49.782 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:16:49.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:16:49.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:16:49.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:17:09.785 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:17:09.821 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:17:09.821 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:17:09.821 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:17:09.821 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:17:09.821 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:17:09.821 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:17:09.821 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:17:09.824 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:17:09.824 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:17:09.825 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:17:09.825 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:17:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:17:09.825 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:17:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:17:09.825 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:17:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:17:09.825 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:17:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:17:09.825 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:17:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:17:09.825 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:17:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:17:09.825 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:17:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:17:09.825 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:17:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:17:09.825 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:17:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:17:09.825 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:17:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:17:09.825 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:17:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:17:09.825 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:17:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:17:09.825 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:17:09.825 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:17:09.825 139328 ERROR oslo_service.periodic_task >2018-06-25 01:17:39.697 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:17:39.698 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:17:39.704 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:17:39.705 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:17:39.740 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:17:39.740 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:17:39.740 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:17:39.740 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:17:39.740 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:17:39.740 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:17:39.740 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:17:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:17:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:17:39.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:17:49.745 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:17:49.777 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:17:49.777 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:17:49.777 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:17:49.777 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:17:49.777 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:17:49.777 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:17:49.777 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:17:49.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:17:49.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:17:49.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:18:09.780 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:18:09.815 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:18:09.815 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:18:09.815 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:18:09.815 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:18:09.815 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:18:09.815 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:18:09.815 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:18:09.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:18:09.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:18:09.817 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:18:09.817 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:18:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:18:09.817 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:18:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:18:09.817 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:18:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:18:09.817 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:18:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:18:09.817 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:18:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:18:09.817 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:18:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:18:09.817 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:18:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:18:09.817 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:18:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:18:09.817 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:18:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:18:09.817 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:18:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:18:09.817 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:18:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:18:09.817 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:18:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:18:09.817 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:18:09.817 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:18:09.817 139328 ERROR oslo_service.periodic_task >2018-06-25 01:18:39.699 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:18:39.700 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:18:39.707 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:18:39.707 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:18:39.740 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:18:39.740 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:18:39.740 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:18:39.740 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:18:39.740 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:18:39.740 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:18:39.740 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:18:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:18:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:18:39.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:18:49.744 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:18:49.782 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:18:49.782 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:18:49.782 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:18:49.782 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:18:49.782 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:18:49.782 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:18:49.782 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:18:49.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:18:49.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:18:49.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:19:09.786 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:19:09.820 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:19:09.820 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:19:09.820 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:19:09.820 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:19:09.820 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:19:09.820 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:19:09.820 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:19:09.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:19:09.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:19:09.824 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:19:09.824 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:19:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:19:09.824 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:19:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:19:09.824 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:19:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:19:09.824 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:19:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:19:09.824 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:19:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:19:09.824 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:19:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:19:09.824 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:19:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:19:09.824 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:19:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:19:09.824 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:19:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:19:09.824 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:19:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:19:09.824 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:19:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:19:09.824 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:19:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:19:09.824 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:19:09.824 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:19:09.824 139328 ERROR oslo_service.periodic_task >2018-06-25 01:19:39.695 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:19:39.696 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:19:39.702 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:19:39.703 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:19:39.739 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:19:39.739 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:19:39.739 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:19:39.739 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:19:39.739 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:19:39.739 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:19:39.739 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:19:39.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:19:39.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:19:39.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:19:49.742 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:19:49.784 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:19:49.784 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:19:49.784 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:19:49.784 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:19:49.784 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:19:49.784 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:19:49.784 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:19:49.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:19:49.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.083 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:19:49.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:20:09.787 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:20:09.819 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:20:09.819 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:20:09.819 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:20:09.819 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:20:09.819 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:20:09.819 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:20:09.819 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:20:09.821 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:20:09.822 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.119 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:20:09.822 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:20:09.822 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:20:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:20:09.822 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:20:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:20:09.822 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:20:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:20:09.822 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:20:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:20:09.822 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:20:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:20:09.822 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:20:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:20:09.822 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:20:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:20:09.822 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:20:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:20:09.822 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:20:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:20:09.822 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:20:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:20:09.822 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:20:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:20:09.822 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:20:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:20:09.822 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:20:09.822 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:20:09.822 139328 ERROR oslo_service.periodic_task >2018-06-25 01:20:39.694 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:20:39.695 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:20:39.703 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:20:39.703 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:20:39.736 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:20:39.736 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:20:39.736 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:20:39.736 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:20:39.736 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:20:39.736 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:20:39.736 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:20:39.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:20:39.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:20:39.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:20:49.739 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:20:49.772 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:20:49.772 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:20:49.772 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:20:49.772 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:20:49.772 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:20:49.772 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:20:49.772 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:20:49.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:20:49.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:20:49.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:21:09.778 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:21:09.812 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:21:09.812 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:21:09.812 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:21:09.812 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:21:09.812 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:21:09.812 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:21:09.812 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:21:09.814 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:21:09.814 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:21:09.814 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:21:09.814 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:21:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:21:09.814 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:21:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:21:09.814 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:21:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:21:09.814 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:21:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:21:09.814 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:21:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:21:09.814 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:21:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:21:09.814 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:21:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:21:09.814 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:21:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:21:09.814 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:21:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:21:09.814 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:21:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:21:09.814 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:21:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:21:09.814 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:21:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:21:09.814 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:21:09.814 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:21:09.814 139328 ERROR oslo_service.periodic_task >2018-06-25 01:21:39.696 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:21:39.697 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:21:39.730 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:21:39.731 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:21:39.776 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:21:39.776 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:21:39.776 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:21:39.776 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:21:39.776 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:21:39.776 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:21:39.776 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:21:39.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:21:39.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.047 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:21:39.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:21:49.779 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:21:49.817 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:21:49.817 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:21:49.817 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:21:49.817 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:21:49.817 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:21:49.817 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:21:49.817 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:21:49.819 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:21:49.820 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.089 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:21:49.820 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:22:09.821 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:22:09.864 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:22:09.864 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:22:09.864 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:22:09.864 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:22:09.864 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:22:09.864 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:22:09.864 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:22:09.866 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:22:09.867 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.136 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:22:09.867 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:22:09.867 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:22:09.867 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:22:09.867 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:22:09.867 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:22:09.867 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:22:09.867 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:22:09.867 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:22:09.867 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:22:09.867 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:22:09.867 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:22:09.867 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:22:09.867 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:22:09.867 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:22:09.867 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:22:09.867 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:22:09.867 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:22:09.867 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:22:09.867 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:22:09.867 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:22:09.867 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:22:09.867 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:22:09.867 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:22:09.867 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:22:09.867 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:22:09.867 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:22:09.867 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:22:09.867 139328 ERROR oslo_service.periodic_task >2018-06-25 01:22:39.699 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:22:39.700 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:22:39.707 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:22:39.708 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:22:39.745 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:22:39.745 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:22:39.745 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:22:39.745 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:22:39.745 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:22:39.745 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:22:39.745 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:22:39.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:22:39.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.041 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:22:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:22:49.750 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:22:49.785 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:22:49.785 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:22:49.785 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:22:49.785 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:22:49.785 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:22:49.785 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:22:49.785 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:22:49.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:22:49.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.081 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:22:49.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:23:09.790 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:23:09.824 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:23:09.824 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:23:09.824 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:23:09.824 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:23:09.824 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:23:09.824 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:23:09.824 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:23:09.826 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:23:09.827 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.119 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:23:09.827 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:23:09.827 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:23:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:23:09.827 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:23:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:23:09.827 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:23:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:23:09.827 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:23:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:23:09.827 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:23:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:23:09.827 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:23:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:23:09.827 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:23:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:23:09.827 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:23:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:23:09.827 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:23:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:23:09.827 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:23:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:23:09.827 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:23:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:23:09.827 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:23:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:23:09.827 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:23:09.827 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:23:09.827 139328 ERROR oslo_service.periodic_task >2018-06-25 01:23:39.698 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:23:39.699 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:23:39.707 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:23:39.707 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:23:39.742 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:23:39.742 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:23:39.742 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:23:39.742 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:23:39.742 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:23:39.742 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:23:39.742 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:23:39.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:23:39.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:23:39.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:23:49.745 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:23:49.779 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:23:49.779 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:23:49.779 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:23:49.779 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:23:49.779 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:23:49.779 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:23:49.779 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:23:49.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:23:49.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:23:49.783 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:24:09.783 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:24:09.824 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:24:09.824 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:24:09.824 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:24:09.824 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:24:09.824 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:24:09.824 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:24:09.824 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:24:09.827 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:24:09.828 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.12 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:24:09.828 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:24:09.828 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:24:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:24:09.828 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:24:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:24:09.828 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:24:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:24:09.828 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:24:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:24:09.828 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:24:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:24:09.828 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:24:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:24:09.828 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:24:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:24:09.828 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:24:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:24:09.828 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:24:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:24:09.828 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:24:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:24:09.828 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:24:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:24:09.828 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:24:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:24:09.828 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:24:09.828 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:24:09.828 139328 ERROR oslo_service.periodic_task >2018-06-25 01:24:39.700 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:24:39.702 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:24:39.710 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:24:39.710 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:24:39.749 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:24:39.749 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:24:39.749 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:24:39.749 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:24:39.749 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:24:39.749 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:24:39.749 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:24:39.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:24:39.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.041 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:24:39.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:24:49.752 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:24:49.786 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:24:49.786 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:24:49.786 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:24:49.786 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:24:49.786 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:24:49.786 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:24:49.786 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:24:49.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:24:49.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:24:49.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:25:09.790 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:25:09.830 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:25:09.830 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:25:09.830 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:25:09.830 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:25:09.830 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:25:09.830 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:25:09.830 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:25:09.832 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:25:09.832 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.122 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:25:09.833 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:25:09.833 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:25:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:25:09.833 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:25:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:25:09.833 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:25:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:25:09.833 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:25:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:25:09.833 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:25:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:25:09.833 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:25:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:25:09.833 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:25:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:25:09.833 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:25:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:25:09.833 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:25:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:25:09.833 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:25:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:25:09.833 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:25:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:25:09.833 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:25:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:25:09.833 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:25:09.833 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:25:09.833 139328 ERROR oslo_service.periodic_task >2018-06-25 01:25:39.704 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:25:39.705 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:25:39.712 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:25:39.713 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:25:39.761 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:25:39.761 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:25:39.761 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:25:39.761 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:25:39.761 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:25:39.761 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:25:39.761 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:25:39.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:25:39.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.051 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:25:39.764 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:25:49.764 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:25:49.798 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:25:49.798 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:25:49.798 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:25:49.798 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:25:49.798 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:25:49.798 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:25:49.798 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:25:49.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:25:49.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.087 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:25:49.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:26:09.801 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:26:09.838 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:26:09.838 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:26:09.838 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:26:09.838 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:26:09.838 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:26:09.838 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:26:09.838 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:26:09.842 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:26:09.843 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.13 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:26:09.843 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:26:09.843 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:26:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:26:09.843 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:26:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:26:09.843 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:26:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:26:09.843 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:26:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:26:09.843 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:26:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:26:09.843 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:26:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:26:09.843 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:26:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:26:09.843 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:26:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:26:09.843 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:26:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:26:09.843 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:26:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:26:09.843 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:26:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:26:09.843 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:26:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:26:09.843 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:26:09.843 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:26:09.843 139328 ERROR oslo_service.periodic_task >2018-06-25 01:26:39.705 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:26:39.707 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:26:39.714 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:26:39.714 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:26:39.750 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:26:39.750 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:26:39.750 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:26:39.750 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:26:39.750 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:26:39.750 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:26:39.750 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:26:39.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:26:39.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:26:39.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:26:49.753 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:26:49.789 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:26:49.789 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:26:49.789 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:26:49.789 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:26:49.789 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:26:49.789 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:26:49.789 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:26:49.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:26:49.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:26:49.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:27:09.792 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:27:09.827 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:27:09.827 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:27:09.827 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:27:09.827 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:27:09.827 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:27:09.827 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:27:09.827 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:27:09.830 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:27:09.831 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:27:09.831 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:27:09.831 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:27:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:27:09.831 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:27:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:27:09.831 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:27:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:27:09.831 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:27:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:27:09.831 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:27:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:27:09.831 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:27:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:27:09.831 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:27:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:27:09.831 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:27:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:27:09.831 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:27:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:27:09.831 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:27:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:27:09.831 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:27:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:27:09.831 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:27:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:27:09.831 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:27:09.831 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:27:09.831 139328 ERROR oslo_service.periodic_task >2018-06-25 01:27:39.703 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:27:39.704 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:27:39.711 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:27:39.711 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:27:39.744 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:27:39.744 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:27:39.744 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:27:39.744 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:27:39.744 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:27:39.744 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:27:39.744 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:27:39.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:27:39.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:27:39.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:27:49.747 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:27:49.780 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:27:49.780 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:27:49.780 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:27:49.780 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:27:49.780 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:27:49.780 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:27:49.780 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:27:49.781 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:27:49.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.07 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:27:49.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:28:09.783 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:28:09.815 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:28:09.815 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:28:09.815 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:28:09.815 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:28:09.815 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:28:09.815 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:28:09.815 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:28:09.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:28:09.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.105 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:28:09.817 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:28:09.817 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:28:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:28:09.817 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:28:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:28:09.817 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:28:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:28:09.817 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:28:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:28:09.817 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:28:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:28:09.817 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:28:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:28:09.817 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:28:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:28:09.817 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:28:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:28:09.817 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:28:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:28:09.817 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:28:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:28:09.817 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:28:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:28:09.817 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:28:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:28:09.817 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:28:09.817 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:28:09.817 139328 ERROR oslo_service.periodic_task >2018-06-25 01:28:39.699 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:28:39.700 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:28:39.706 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:28:39.707 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:28:39.742 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:28:39.742 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:28:39.742 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:28:39.742 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:28:39.742 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:28:39.742 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:28:39.742 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:28:39.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:28:39.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:28:39.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:28:49.745 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:28:49.779 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:28:49.779 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:28:49.779 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:28:49.779 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:28:49.779 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:28:49.779 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:28:49.779 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:28:49.781 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:28:49.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:28:49.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:29:09.782 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:29:09.815 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:29:09.815 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:29:09.815 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:29:09.815 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:29:09.815 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:29:09.815 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:29:09.815 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:29:09.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:29:09.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:29:09.818 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:29:09.818 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:29:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:29:09.818 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:29:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:29:09.818 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:29:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:29:09.818 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:29:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:29:09.818 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:29:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:29:09.818 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:29:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:29:09.818 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:29:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:29:09.818 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:29:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:29:09.818 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:29:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:29:09.818 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:29:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:29:09.818 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:29:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:29:09.818 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:29:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:29:09.818 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:29:09.818 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:29:09.818 139328 ERROR oslo_service.periodic_task >2018-06-25 01:29:39.699 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:29:39.701 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:29:39.707 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:29:39.708 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:29:39.741 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:29:39.741 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:29:39.741 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:29:39.741 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:29:39.741 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:29:39.741 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:29:39.741 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:29:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:29:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:29:39.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:29:49.744 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:29:49.782 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:29:49.782 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:29:49.782 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:29:49.782 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:29:49.782 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:29:49.782 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:29:49.782 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:29:49.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:29:49.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:29:49.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:30:09.785 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:30:09.826 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:30:09.826 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:30:09.826 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:30:09.826 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:30:09.826 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:30:09.826 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:30:09.826 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:30:09.828 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:30:09.828 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.12 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:30:09.828 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:30:09.828 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:30:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:30:09.828 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:30:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:30:09.828 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:30:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:30:09.828 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:30:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:30:09.828 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:30:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:30:09.828 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:30:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:30:09.828 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:30:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:30:09.828 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:30:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:30:09.828 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:30:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:30:09.828 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:30:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:30:09.828 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:30:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:30:09.828 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:30:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:30:09.828 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:30:09.828 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:30:09.828 139328 ERROR oslo_service.periodic_task >2018-06-25 01:30:39.700 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:30:39.701 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:30:39.708 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:30:39.709 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:30:39.741 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:30:39.741 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:30:39.741 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:30:39.741 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:30:39.741 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:30:39.741 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:30:39.741 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:30:39.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:30:39.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:30:39.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:30:49.745 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:30:49.788 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:30:49.788 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:30:49.788 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:30:49.788 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:30:49.788 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:30:49.788 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:30:49.788 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:30:49.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:30:49.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.081 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:30:49.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:31:09.791 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:31:09.830 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:31:09.830 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:31:09.830 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:31:09.830 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:31:09.830 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:31:09.830 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:31:09.830 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:31:09.833 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:31:09.833 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.124 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:31:09.834 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:31:09.834 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:31:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:31:09.834 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:31:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:31:09.834 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:31:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:31:09.834 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:31:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:31:09.834 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:31:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:31:09.834 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:31:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:31:09.834 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:31:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:31:09.834 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:31:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:31:09.834 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:31:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:31:09.834 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:31:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:31:09.834 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:31:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:31:09.834 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:31:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:31:09.834 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:31:09.834 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:31:09.834 139328 ERROR oslo_service.periodic_task >2018-06-25 01:31:39.705 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:31:39.706 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:31:39.714 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:31:39.715 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:31:39.767 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:31:39.767 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:31:39.767 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:31:39.767 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:31:39.767 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:31:39.767 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:31:39.767 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:31:39.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:31:39.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.055 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:31:39.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:31:49.771 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:31:49.806 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:31:49.806 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:31:49.806 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:31:49.806 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:31:49.806 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:31:49.806 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:31:49.806 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:31:49.809 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:31:49.809 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.094 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:31:49.809 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:32:09.810 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:32:09.845 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:32:09.845 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:32:09.845 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:32:09.845 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:32:09.845 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:32:09.845 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:32:09.845 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:32:09.847 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:32:09.847 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.132 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:32:09.848 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:32:09.848 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:32:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:32:09.848 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:32:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:32:09.848 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:32:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:32:09.848 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:32:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:32:09.848 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:32:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:32:09.848 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:32:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:32:09.848 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:32:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:32:09.848 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:32:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:32:09.848 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:32:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:32:09.848 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:32:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:32:09.848 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:32:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:32:09.848 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:32:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:32:09.848 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:32:09.848 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:32:09.848 139328 ERROR oslo_service.periodic_task >2018-06-25 01:32:39.709 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:32:39.710 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:32:39.718 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:32:39.718 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:32:39.752 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:32:39.752 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:32:39.752 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:32:39.752 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:32:39.752 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:32:39.752 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:32:39.752 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:32:39.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:32:39.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:32:39.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:32:49.755 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:32:49.791 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:32:49.791 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:32:49.791 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:32:49.791 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:32:49.791 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:32:49.791 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:32:49.791 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:32:49.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:32:49.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:32:49.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:33:09.794 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:33:09.827 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:33:09.827 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:33:09.827 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:33:09.827 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:33:09.827 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:33:09.827 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:33:09.827 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:33:09.829 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:33:09.829 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:33:09.830 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:33:09.830 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:33:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:33:09.830 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:33:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:33:09.830 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:33:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:33:09.830 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:33:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:33:09.830 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:33:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:33:09.830 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:33:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:33:09.830 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:33:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:33:09.830 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:33:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:33:09.830 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:33:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:33:09.830 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:33:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:33:09.830 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:33:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:33:09.830 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:33:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:33:09.830 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:33:09.830 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:33:09.830 139328 ERROR oslo_service.periodic_task >2018-06-25 01:33:39.711 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:33:39.712 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:33:39.719 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:33:39.720 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:33:39.758 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:33:39.758 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:33:39.758 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:33:39.758 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:33:39.758 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:33:39.758 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:33:39.758 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:33:39.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:33:39.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.041 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:33:39.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:33:49.762 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:33:49.795 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:33:49.795 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:33:49.795 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:33:49.795 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:33:49.795 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:33:49.795 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:33:49.795 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:33:49.797 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:33:49.798 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:33:49.798 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:34:09.798 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:34:09.833 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:34:09.833 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:34:09.833 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:34:09.833 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:34:09.833 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:34:09.833 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:34:09.833 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:34:09.836 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:34:09.836 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:34:09.836 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:34:09.836 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:34:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:34:09.836 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:34:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:34:09.836 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:34:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:34:09.836 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:34:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:34:09.836 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:34:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:34:09.836 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:34:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:34:09.836 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:34:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:34:09.836 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:34:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:34:09.836 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:34:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:34:09.836 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:34:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:34:09.836 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:34:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:34:09.836 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:34:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:34:09.836 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:34:09.836 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:34:09.836 139328 ERROR oslo_service.periodic_task >2018-06-25 01:34:39.708 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:34:39.709 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:34:39.718 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:34:39.719 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:34:39.756 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:34:39.756 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:34:39.756 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:34:39.756 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:34:39.756 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:34:39.756 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:34:39.756 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:34:39.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:34:39.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:34:39.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:34:49.759 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:34:49.792 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:34:49.792 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:34:49.792 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:34:49.792 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:34:49.792 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:34:49.792 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:34:49.792 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:34:49.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:34:49.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:34:49.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:35:09.795 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:35:09.826 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:35:09.826 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:35:09.826 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:35:09.826 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:35:09.826 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:35:09.826 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:35:09.826 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:35:09.828 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:35:09.828 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:35:09.829 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:35:09.829 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:35:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:35:09.829 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:35:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:35:09.829 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:35:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:35:09.829 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:35:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:35:09.829 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:35:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:35:09.829 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:35:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:35:09.829 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:35:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:35:09.829 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:35:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:35:09.829 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:35:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:35:09.829 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:35:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:35:09.829 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:35:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:35:09.829 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:35:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:35:09.829 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:35:09.829 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:35:09.829 139328 ERROR oslo_service.periodic_task >2018-06-25 01:35:39.710 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:35:39.711 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:35:39.718 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:35:39.719 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:35:39.758 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:35:39.758 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:35:39.758 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:35:39.758 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:35:39.758 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:35:39.758 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:35:39.758 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:35:39.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:35:39.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:35:39.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:35:49.761 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:35:49.796 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:35:49.796 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:35:49.796 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:35:49.796 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:35:49.796 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:35:49.796 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:35:49.796 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:35:49.798 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:35:49.798 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:35:49.799 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:36:09.799 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:36:09.833 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:36:09.833 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:36:09.833 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:36:09.833 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:36:09.833 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:36:09.833 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:36:09.833 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:36:09.835 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:36:09.835 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:36:09.836 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:36:09.836 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:36:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:36:09.836 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:36:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:36:09.836 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:36:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:36:09.836 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:36:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:36:09.836 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:36:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:36:09.836 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:36:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:36:09.836 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:36:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:36:09.836 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:36:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:36:09.836 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:36:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:36:09.836 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:36:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:36:09.836 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:36:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:36:09.836 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:36:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:36:09.836 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:36:09.836 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:36:09.836 139328 ERROR oslo_service.periodic_task >2018-06-25 01:36:39.707 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:36:39.709 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:36:39.716 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:36:39.716 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:36:39.755 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:36:39.755 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:36:39.755 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:36:39.755 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:36:39.755 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:36:39.755 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:36:39.755 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:36:39.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:36:39.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.041 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:36:39.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:36:49.759 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:36:49.794 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:36:49.794 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:36:49.794 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:36:49.794 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:36:49.794 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:36:49.794 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:36:49.794 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:36:49.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:36:49.797 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.08 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:36:49.797 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:37:09.798 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:37:09.835 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:37:09.835 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:37:09.835 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:37:09.835 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:37:09.835 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:37:09.835 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:37:09.835 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:37:09.837 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:37:09.838 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.121 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:37:09.838 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:37:09.838 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:37:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:37:09.838 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:37:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:37:09.838 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:37:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:37:09.838 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:37:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:37:09.838 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:37:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:37:09.838 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:37:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:37:09.838 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:37:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:37:09.838 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:37:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:37:09.838 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:37:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:37:09.838 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:37:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:37:09.838 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:37:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:37:09.838 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:37:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:37:09.838 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:37:09.838 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:37:09.838 139328 ERROR oslo_service.periodic_task >2018-06-25 01:37:39.710 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:37:39.711 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:37:39.718 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:37:39.719 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:37:39.752 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:37:39.752 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:37:39.752 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:37:39.752 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:37:39.752 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:37:39.752 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:37:39.752 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:37:39.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:37:39.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:37:39.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:37:49.755 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:37:49.788 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:37:49.788 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:37:49.788 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:37:49.788 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:37:49.788 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:37:49.788 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:37:49.788 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:37:49.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:37:49.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:37:49.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:38:09.792 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:38:09.825 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:38:09.825 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:38:09.825 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:38:09.825 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:38:09.825 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:38:09.825 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:38:09.825 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:38:09.826 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:38:09.827 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:38:09.827 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:38:09.827 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:38:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:38:09.827 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:38:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:38:09.827 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:38:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:38:09.827 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:38:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:38:09.827 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:38:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:38:09.827 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:38:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:38:09.827 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:38:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:38:09.827 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:38:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:38:09.827 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:38:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:38:09.827 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:38:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:38:09.827 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:38:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:38:09.827 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:38:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:38:09.827 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:38:09.827 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:38:09.827 139328 ERROR oslo_service.periodic_task >2018-06-25 01:38:39.708 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:38:39.710 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:38:39.717 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:38:39.718 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:38:39.751 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:38:39.751 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:38:39.751 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:38:39.751 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:38:39.751 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:38:39.751 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:38:39.751 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:38:39.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:38:39.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:38:39.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:38:49.755 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:38:49.788 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:38:49.788 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:38:49.788 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:38:49.788 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:38:49.788 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:38:49.788 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:38:49.788 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:38:49.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:38:49.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:38:49.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:39:09.791 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:39:09.827 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:39:09.827 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:39:09.827 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:39:09.827 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:39:09.827 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:39:09.827 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:39:09.827 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:39:09.829 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:39:09.829 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:39:09.829 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:39:09.829 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:39:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:39:09.829 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:39:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:39:09.829 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:39:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:39:09.829 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:39:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:39:09.829 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:39:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:39:09.829 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:39:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:39:09.829 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:39:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:39:09.829 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:39:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:39:09.829 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:39:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:39:09.829 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:39:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:39:09.829 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:39:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:39:09.829 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:39:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:39:09.829 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:39:09.829 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:39:09.829 139328 ERROR oslo_service.periodic_task >2018-06-25 01:39:39.711 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:39:39.712 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:39:39.718 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:39:39.719 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:39:39.752 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:39:39.752 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:39:39.752 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:39:39.752 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:39:39.752 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:39:39.752 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:39:39.752 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:39:39.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:39:39.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:39:39.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:39:49.755 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:39:49.787 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:39:49.787 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:39:49.787 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:39:49.787 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:39:49.787 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:39:49.787 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:39:49.787 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:39:49.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:39:49.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:39:49.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:40:09.791 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:40:09.825 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:40:09.825 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:40:09.825 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:40:09.825 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:40:09.825 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:40:09.825 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:40:09.825 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:40:09.827 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:40:09.827 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:40:09.828 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:40:09.828 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:40:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:40:09.828 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:40:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:40:09.828 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:40:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:40:09.828 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:40:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:40:09.828 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:40:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:40:09.828 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:40:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:40:09.828 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:40:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:40:09.828 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:40:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:40:09.828 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:40:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:40:09.828 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:40:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:40:09.828 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:40:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:40:09.828 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:40:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:40:09.828 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:40:09.828 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:40:09.828 139328 ERROR oslo_service.periodic_task >2018-06-25 01:40:39.709 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:40:39.710 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:40:39.720 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:40:39.720 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:40:39.754 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:40:39.754 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:40:39.754 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:40:39.754 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:40:39.754 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:40:39.754 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:40:39.754 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:40:39.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:40:39.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:40:39.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:40:49.757 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:40:49.790 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:40:49.790 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:40:49.790 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:40:49.790 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:40:49.790 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:40:49.790 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:40:49.790 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:40:49.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:40:49.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:40:49.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:41:09.793 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:41:09.826 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:41:09.826 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:41:09.826 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:41:09.826 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:41:09.826 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:41:09.826 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:41:09.826 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:41:09.828 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:41:09.828 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.107 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:41:09.829 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:41:09.829 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:41:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:41:09.829 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:41:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:41:09.829 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:41:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:41:09.829 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:41:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:41:09.829 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:41:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:41:09.829 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:41:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:41:09.829 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:41:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:41:09.829 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:41:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:41:09.829 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:41:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:41:09.829 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:41:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:41:09.829 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:41:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:41:09.829 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:41:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:41:09.829 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:41:09.829 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:41:09.829 139328 ERROR oslo_service.periodic_task >2018-06-25 01:41:39.711 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:41:39.712 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:41:39.718 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:41:39.718 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:41:39.756 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:41:39.756 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:41:39.756 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:41:39.756 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:41:39.756 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:41:39.756 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:41:39.756 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:41:39.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:41:39.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:41:39.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:41:49.759 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:41:49.792 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:41:49.792 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:41:49.792 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:41:49.792 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:41:49.792 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:41:49.792 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:41:49.792 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:41:49.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:41:49.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:41:49.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:42:09.796 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:42:09.831 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:42:09.831 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:42:09.831 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:42:09.831 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:42:09.831 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:42:09.831 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:42:09.831 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:42:09.834 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:42:09.834 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:42:09.835 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:42:09.835 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:42:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:42:09.835 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:42:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:42:09.835 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:42:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:42:09.835 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:42:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:42:09.835 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:42:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:42:09.835 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:42:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:42:09.835 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:42:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:42:09.835 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:42:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:42:09.835 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:42:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:42:09.835 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:42:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:42:09.835 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:42:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:42:09.835 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:42:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:42:09.835 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:42:09.835 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:42:09.835 139328 ERROR oslo_service.periodic_task >2018-06-25 01:42:39.707 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:42:39.707 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:42:39.717 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:42:39.718 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:42:39.753 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:42:39.753 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:42:39.753 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:42:39.753 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:42:39.753 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:42:39.753 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:42:39.753 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:42:39.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:42:39.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:42:39.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:42:49.756 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:42:49.790 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:42:49.790 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:42:49.790 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:42:49.790 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:42:49.790 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:42:49.790 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:42:49.790 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:42:49.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:42:49.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:42:49.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:43:09.794 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:43:09.825 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:43:09.825 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:43:09.825 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:43:09.825 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:43:09.825 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:43:09.825 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:43:09.825 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:43:09.827 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:43:09.827 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:43:09.827 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:43:09.827 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:43:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:43:09.827 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:43:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:43:09.827 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:43:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:43:09.827 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:43:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:43:09.827 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:43:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:43:09.827 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:43:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:43:09.827 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:43:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:43:09.827 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:43:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:43:09.827 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:43:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:43:09.827 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:43:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:43:09.827 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:43:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:43:09.827 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:43:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:43:09.827 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:43:09.827 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:43:09.827 139328 ERROR oslo_service.periodic_task >2018-06-25 01:43:39.709 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:43:39.710 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:43:39.717 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:43:39.717 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:43:39.757 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:43:39.757 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:43:39.757 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:43:39.757 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:43:39.757 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:43:39.757 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:43:39.757 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:43:39.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:43:39.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.043 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:43:39.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:43:49.763 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:43:49.796 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:43:49.796 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:43:49.796 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:43:49.796 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:43:49.796 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:43:49.796 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:43:49.796 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:43:49.798 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:43:49.799 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.081 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:43:49.799 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:44:09.799 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:44:09.832 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:44:09.832 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:44:09.832 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:44:09.832 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:44:09.832 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:44:09.832 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:44:09.832 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:44:09.834 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:44:09.834 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:44:09.835 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:44:09.835 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:44:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:44:09.835 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:44:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:44:09.835 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:44:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:44:09.835 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:44:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:44:09.835 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:44:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:44:09.835 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:44:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:44:09.835 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:44:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:44:09.835 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:44:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:44:09.835 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:44:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:44:09.835 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:44:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:44:09.835 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:44:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:44:09.835 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:44:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:44:09.835 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:44:09.835 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:44:09.835 139328 ERROR oslo_service.periodic_task >2018-06-25 01:44:39.706 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:44:39.707 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:45:39.705 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:45:39.705 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:45:39.712 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:45:39.712 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:45:39.745 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:45:39.745 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:45:39.745 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:45:39.745 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:45:39.745 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:45:39.745 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:45:39.745 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:45:39.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:45:39.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:45:39.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:45:49.748 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:45:49.783 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:45:49.783 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:45:49.783 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:45:49.783 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:45:49.783 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:45:49.783 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:45:49.783 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:45:49.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:45:49.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:45:49.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:46:09.786 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:46:09.821 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:46:09.821 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:46:09.821 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:46:09.821 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:46:09.821 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:46:09.821 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:46:09.821 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:46:09.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:46:09.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:46:09.824 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:46:09.824 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:46:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:46:09.824 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:46:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:46:09.824 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:46:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:46:09.824 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:46:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:46:09.824 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:46:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:46:09.824 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:46:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:46:09.824 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:46:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:46:09.824 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:46:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:46:09.824 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:46:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:46:09.824 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:46:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:46:09.824 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:46:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:46:09.824 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:46:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:46:09.824 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:46:09.824 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:46:09.824 139328 ERROR oslo_service.periodic_task >2018-06-25 01:46:39.705 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:46:39.706 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:46:39.713 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:46:39.713 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:46:39.748 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:46:39.748 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:46:39.748 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:46:39.748 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:46:39.748 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:46:39.748 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:46:39.748 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:46:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:46:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:46:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:46:49.751 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:46:49.784 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:46:49.784 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:46:49.784 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:46:49.784 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:46:49.784 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:46:49.784 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:46:49.784 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:46:49.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:46:49.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:46:49.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:47:09.787 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:47:09.822 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:47:09.822 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:47:09.822 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:47:09.822 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:47:09.822 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:47:09.822 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:47:09.822 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:47:09.824 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:47:09.824 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:47:09.825 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:47:09.825 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:47:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:47:09.825 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:47:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:47:09.825 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:47:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:47:09.825 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:47:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:47:09.825 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:47:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:47:09.825 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:47:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:47:09.825 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:47:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:47:09.825 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:47:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:47:09.825 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:47:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:47:09.825 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:47:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:47:09.825 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:47:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:47:09.825 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:47:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:47:09.825 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:47:09.825 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:47:09.825 139328 ERROR oslo_service.periodic_task >2018-06-25 01:47:39.707 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:47:39.707 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:47:39.715 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:47:39.716 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:47:39.748 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:47:39.748 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:47:39.748 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:47:39.748 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:47:39.748 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:47:39.748 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:47:39.748 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:47:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:47:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:47:39.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:47:49.751 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:47:49.784 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:47:49.784 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:47:49.784 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:47:49.784 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:47:49.784 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:47:49.784 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:47:49.784 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:47:49.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:47:49.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:47:49.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:48:09.788 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:48:09.819 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:48:09.819 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:48:09.819 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:48:09.819 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:48:09.819 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:48:09.819 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:48:09.819 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:48:09.821 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:48:09.822 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.106 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:48:09.822 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:48:09.822 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:48:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:48:09.822 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:48:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:48:09.822 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:48:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:48:09.822 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:48:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:48:09.822 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:48:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:48:09.822 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:48:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:48:09.822 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:48:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:48:09.822 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:48:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:48:09.822 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:48:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:48:09.822 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:48:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:48:09.822 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:48:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:48:09.822 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:48:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:48:09.822 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:48:09.822 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:48:09.822 139328 ERROR oslo_service.periodic_task >2018-06-25 01:48:39.703 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:48:39.704 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:48:39.710 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:48:39.711 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:48:39.745 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:48:39.745 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:48:39.745 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:48:39.745 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:48:39.745 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:48:39.745 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:48:39.745 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:48:39.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:48:39.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:48:39.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:48:49.748 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:48:49.780 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:48:49.780 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:48:49.780 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:48:49.780 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:48:49.780 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:48:49.780 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:48:49.780 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:48:49.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:48:49.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:48:49.783 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:49:09.783 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:49:09.817 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:49:09.817 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:49:09.817 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:49:09.817 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:49:09.817 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:49:09.817 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:49:09.817 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:49:09.820 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:49:09.820 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:49:09.820 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:49:09.820 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:49:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:49:09.820 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:49:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:49:09.820 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:49:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:49:09.820 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:49:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:49:09.820 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:49:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:49:09.820 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:49:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:49:09.820 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:49:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:49:09.820 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:49:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:49:09.820 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:49:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:49:09.820 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:49:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:49:09.820 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:49:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:49:09.820 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:49:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:49:09.820 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:49:09.820 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:49:09.820 139328 ERROR oslo_service.periodic_task >2018-06-25 01:49:39.702 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:49:39.703 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:49:39.711 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:49:39.712 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:49:39.746 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:49:39.746 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:49:39.746 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:49:39.746 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:49:39.746 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:49:39.746 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:49:39.746 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:49:39.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:49:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:49:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:49:49.751 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:49:49.784 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:49:49.784 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:49:49.784 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:49:49.784 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:49:49.784 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:49:49.784 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:49:49.784 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:49:49.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:49:49.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:49:49.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:50:09.789 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:50:09.822 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:50:09.822 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:50:09.822 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:50:09.822 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:50:09.822 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:50:09.822 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:50:09.822 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:50:09.825 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:50:09.825 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:50:09.826 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:50:09.826 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:50:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:50:09.826 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:50:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:50:09.826 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:50:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:50:09.826 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:50:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:50:09.826 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:50:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:50:09.826 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:50:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:50:09.826 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:50:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:50:09.826 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:50:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:50:09.826 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:50:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:50:09.826 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:50:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:50:09.826 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:50:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:50:09.826 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:50:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:50:09.826 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:50:09.826 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:50:09.826 139328 ERROR oslo_service.periodic_task >2018-06-25 01:50:39.697 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:50:39.698 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:50:39.710 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:50:39.711 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:50:39.751 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:50:39.751 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:50:39.751 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:50:39.751 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:50:39.751 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:50:39.751 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:50:39.751 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:50:39.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:50:39.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.043 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:50:39.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:50:49.755 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:50:49.785 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:50:49.785 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:50:49.785 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:50:49.785 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:50:49.785 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:50:49.785 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:50:49.785 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:50:49.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:50:49.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:50:49.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:51:09.788 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:51:09.819 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:51:09.819 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:51:09.819 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:51:09.819 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:51:09.819 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:51:09.819 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:51:09.819 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:51:09.821 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:51:09.821 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:51:09.822 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:51:09.822 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:51:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:51:09.822 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:51:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:51:09.822 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:51:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:51:09.822 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:51:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:51:09.822 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:51:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:51:09.822 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:51:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:51:09.822 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:51:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:51:09.822 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:51:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:51:09.822 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:51:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:51:09.822 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:51:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:51:09.822 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:51:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:51:09.822 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:51:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:51:09.822 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:51:09.822 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:51:09.822 139328 ERROR oslo_service.periodic_task >2018-06-25 01:51:39.694 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:51:39.695 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:51:39.703 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:51:39.704 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:51:39.739 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:51:39.739 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:51:39.739 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:51:39.739 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:51:39.739 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:51:39.739 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:51:39.739 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:51:39.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:51:39.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:51:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:51:49.743 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:51:49.775 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:51:49.775 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:51:49.775 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:51:49.775 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:51:49.775 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:51:49.775 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:51:49.775 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:51:49.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:51:49.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:51:49.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:52:09.778 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:52:09.813 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:52:09.813 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:52:09.813 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:52:09.813 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:52:09.813 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:52:09.813 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:52:09.813 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:52:09.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:52:09.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:52:09.817 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:52:09.817 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:52:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:52:09.817 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:52:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:52:09.817 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:52:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:52:09.817 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:52:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:52:09.817 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:52:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:52:09.817 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:52:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:52:09.817 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:52:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:52:09.817 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:52:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:52:09.817 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:52:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:52:09.817 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:52:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:52:09.817 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:52:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:52:09.817 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:52:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:52:09.817 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:52:09.817 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:52:09.817 139328 ERROR oslo_service.periodic_task >2018-06-25 01:52:39.689 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:52:39.690 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:52:39.696 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:52:39.697 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:52:39.728 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:52:39.728 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:52:39.728 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:52:39.728 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:52:39.728 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:52:39.728 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:52:39.728 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:52:39.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:52:39.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:52:39.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:52:49.731 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:52:49.765 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:52:49.765 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:52:49.765 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:52:49.765 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:52:49.765 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:52:49.765 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:52:49.765 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:52:49.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:52:49.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.07 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:52:49.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:53:09.768 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:53:09.802 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:53:09.802 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:53:09.802 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:53:09.802 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:53:09.802 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:53:09.802 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:53:09.802 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:53:09.804 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:53:09.804 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.107 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:53:09.804 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:53:09.804 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:53:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:53:09.804 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:53:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:53:09.804 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:53:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:53:09.804 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:53:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:53:09.804 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:53:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:53:09.804 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:53:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:53:09.804 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:53:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:53:09.804 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:53:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:53:09.804 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:53:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:53:09.804 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:53:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:53:09.804 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:53:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:53:09.804 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:53:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:53:09.804 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:53:09.804 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:53:09.804 139328 ERROR oslo_service.periodic_task >2018-06-25 01:53:39.686 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:53:39.688 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:53:39.695 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:53:39.696 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:53:39.728 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:53:39.728 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:53:39.728 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:53:39.728 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:53:39.728 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:53:39.728 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:53:39.728 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:53:39.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:53:39.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:53:39.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:53:49.732 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:53:49.764 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:53:49.764 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:53:49.764 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:53:49.764 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:53:49.764 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:53:49.764 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:53:49.764 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:53:49.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:53:49.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:53:49.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:54:09.767 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:54:09.802 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:54:09.802 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:54:09.802 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:54:09.802 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:54:09.802 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:54:09.802 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:54:09.802 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:54:09.804 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:54:09.804 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:54:09.804 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:54:09.804 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:54:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:54:09.804 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:54:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:54:09.804 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:54:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:54:09.804 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:54:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:54:09.804 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:54:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:54:09.804 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:54:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:54:09.804 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:54:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:54:09.804 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:54:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:54:09.804 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:54:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:54:09.804 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:54:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:54:09.804 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:54:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:54:09.804 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:54:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:54:09.804 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:54:09.804 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:54:09.804 139328 ERROR oslo_service.periodic_task >2018-06-25 01:54:39.686 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:54:39.687 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:54:39.695 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:54:39.696 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:54:39.739 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:54:39.739 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:54:39.739 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:54:39.739 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:54:39.739 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:54:39.739 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:54:39.739 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:54:39.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:54:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.047 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:54:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:54:49.744 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:54:49.781 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:54:49.781 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:54:49.781 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:54:49.781 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:54:49.781 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:54:49.781 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:54:49.781 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:54:49.783 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:54:49.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.088 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:54:49.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:55:09.785 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:55:09.817 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:55:09.817 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:55:09.817 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:55:09.817 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:55:09.817 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:55:09.817 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:55:09.817 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:55:09.819 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:55:09.819 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.123 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:55:09.820 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:55:09.820 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:55:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:55:09.820 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:55:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:55:09.820 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:55:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:55:09.820 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:55:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:55:09.820 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:55:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:55:09.820 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:55:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:55:09.820 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:55:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:55:09.820 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:55:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:55:09.820 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:55:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:55:09.820 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:55:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:55:09.820 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:55:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:55:09.820 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:55:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:55:09.820 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:55:09.820 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:55:09.820 139328 ERROR oslo_service.periodic_task >2018-06-25 01:55:39.691 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:55:39.693 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:55:39.699 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:55:39.700 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:55:39.738 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:55:39.738 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:55:39.738 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:55:39.738 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:55:39.738 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:55:39.738 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:55:39.738 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:55:39.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:55:39.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.041 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:55:39.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:55:49.741 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:55:49.773 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:55:49.773 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:55:49.773 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:55:49.773 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:55:49.773 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:55:49.773 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:55:49.773 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:55:49.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:55:49.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:55:49.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:56:09.777 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:56:09.811 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:56:09.811 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:56:09.811 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:56:09.811 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:56:09.811 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:56:09.811 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:56:09.811 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:56:09.813 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:56:09.813 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:56:09.813 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:56:09.813 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:56:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:56:09.813 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:56:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:56:09.813 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:56:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:56:09.813 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:56:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:56:09.813 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:56:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:56:09.813 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:56:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:56:09.813 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:56:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:56:09.813 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:56:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:56:09.813 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:56:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:56:09.813 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:56:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:56:09.813 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:56:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:56:09.813 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:56:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:56:09.813 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:56:09.813 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:56:09.813 139328 ERROR oslo_service.periodic_task >2018-06-25 01:56:39.695 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:56:39.696 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:56:39.703 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:56:39.703 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:56:39.737 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:56:39.737 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:56:39.737 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:56:39.737 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:56:39.737 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:56:39.737 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:56:39.737 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:56:39.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:56:39.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:56:39.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:56:49.740 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:56:49.774 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:56:49.774 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:56:49.774 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:56:49.774 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:56:49.774 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:56:49.774 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:56:49.774 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:56:49.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:56:49.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:56:49.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:57:09.777 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:57:09.811 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:57:09.811 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:57:09.811 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:57:09.811 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:57:09.811 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:57:09.811 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:57:09.811 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:57:09.813 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:57:09.813 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:57:09.814 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:57:09.814 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:57:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:57:09.814 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:57:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:57:09.814 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:57:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:57:09.814 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:57:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:57:09.814 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:57:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:57:09.814 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:57:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:57:09.814 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:57:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:57:09.814 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:57:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:57:09.814 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:57:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:57:09.814 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:57:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:57:09.814 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:57:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:57:09.814 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:57:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:57:09.814 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:57:09.814 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:57:09.814 139328 ERROR oslo_service.periodic_task >2018-06-25 01:57:39.695 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:57:39.696 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:57:39.703 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:57:39.704 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:57:39.739 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:57:39.739 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:57:39.739 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:57:39.739 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:57:39.739 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:57:39.739 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:57:39.739 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:57:39.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:57:39.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:57:39.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:57:49.743 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:57:49.791 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:57:49.791 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:57:49.791 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:57:49.791 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:57:49.791 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:57:49.791 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:57:49.791 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:57:49.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:57:49.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.09 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:57:49.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:58:09.795 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:58:09.830 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:58:09.830 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:58:09.830 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:58:09.830 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:58:09.830 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:58:09.830 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:58:09.830 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:58:09.832 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:58:09.832 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.129 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:58:09.833 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:58:09.833 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:58:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:58:09.833 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:58:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:58:09.833 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:58:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:58:09.833 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:58:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:58:09.833 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:58:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:58:09.833 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:58:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:58:09.833 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:58:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:58:09.833 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:58:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:58:09.833 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:58:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:58:09.833 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:58:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:58:09.833 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:58:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:58:09.833 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:58:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:58:09.833 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:58:09.833 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:58:09.833 139328 ERROR oslo_service.periodic_task >2018-06-25 01:58:39.694 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:58:39.696 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:58:39.730 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:58:39.730 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:58:39.730 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:58:39.730 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:58:39.730 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:58:39.730 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:58:39.730 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:58:39.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:58:39.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:58:39.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:58:49.734 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:58:49.765 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:58:49.765 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:58:49.765 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:58:49.765 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:58:49.765 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:58:49.765 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:58:49.765 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:58:49.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:58:49.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:58:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:59:09.768 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:59:09.803 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:59:09.803 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:59:09.803 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:59:09.803 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:59:09.803 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:59:09.803 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:59:09.803 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:59:09.805 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:59:09.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:59:09.806 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:59:09.806 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 01:59:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 01:59:09.806 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 01:59:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 01:59:09.806 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 01:59:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 01:59:09.806 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 01:59:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 01:59:09.806 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 01:59:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 01:59:09.806 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 01:59:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 01:59:09.806 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 01:59:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 01:59:09.806 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 01:59:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 01:59:09.806 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 01:59:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 01:59:09.806 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 01:59:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 01:59:09.806 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 01:59:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 01:59:09.806 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 01:59:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 01:59:09.806 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 01:59:09.806 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 01:59:09.806 139328 ERROR oslo_service.periodic_task >2018-06-25 01:59:39.698 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:59:39.698 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 01:59:39.705 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 01:59:39.705 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:59:39.738 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:59:39.738 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:59:39.738 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:59:39.738 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:59:39.738 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:59:39.738 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:59:39.738 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:59:39.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:59:39.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:59:39.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 01:59:49.742 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 01:59:49.775 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 01:59:49.775 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 01:59:49.775 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 01:59:49.775 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 01:59:49.775 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 01:59:49.775 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 01:59:49.775 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 01:59:49.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 01:59:49.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 01:59:49.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:00:09.778 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:00:09.812 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:00:09.812 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:00:09.812 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:00:09.812 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:00:09.812 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:00:09.812 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:00:09.812 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:00:09.814 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:00:09.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:00:09.815 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:00:09.815 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:00:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:00:09.815 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:00:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:00:09.815 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:00:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:00:09.815 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:00:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:00:09.815 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:00:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:00:09.815 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:00:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:00:09.815 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:00:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:00:09.815 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:00:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:00:09.815 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:00:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:00:09.815 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:00:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:00:09.815 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:00:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:00:09.815 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:00:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:00:09.815 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:00:09.815 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:00:09.815 139328 ERROR oslo_service.periodic_task >2018-06-25 02:00:39.697 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:00:39.698 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:00:39.704 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:00:39.705 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:00:39.737 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:00:39.737 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:00:39.737 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:00:39.737 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:00:39.737 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:00:39.737 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:00:39.737 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:00:39.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:00:39.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:00:39.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:00:49.740 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:00:49.772 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:00:49.772 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:00:49.772 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:00:49.772 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:00:49.772 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:00:49.772 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:00:49.772 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:00:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:00:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.069 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:00:49.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:01:09.775 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:01:09.812 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:01:09.812 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:01:09.812 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:01:09.812 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:01:09.812 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:01:09.812 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:01:09.812 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:01:09.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:01:09.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:01:09.816 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:01:09.816 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:01:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:01:09.816 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:01:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:01:09.816 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:01:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:01:09.816 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:01:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:01:09.816 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:01:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:01:09.816 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:01:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:01:09.816 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:01:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:01:09.816 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:01:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:01:09.816 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:01:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:01:09.816 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:01:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:01:09.816 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:01:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:01:09.816 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:01:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:01:09.816 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:01:09.816 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:01:09.816 139328 ERROR oslo_service.periodic_task >2018-06-25 02:01:39.698 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:01:39.700 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:01:39.707 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:01:39.708 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:01:39.750 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:01:39.750 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:01:39.750 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:01:39.750 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:01:39.750 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:01:39.750 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:01:39.750 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:01:39.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:01:39.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.045 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:01:39.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:01:49.755 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:01:49.790 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:01:49.790 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:01:49.790 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:01:49.790 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:01:49.790 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:01:49.790 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:01:49.790 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:01:49.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:01:49.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.084 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:01:49.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:02:09.793 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:02:09.826 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:02:09.826 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:02:09.826 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:02:09.826 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:02:09.826 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:02:09.826 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:02:09.826 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:02:09.829 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:02:09.829 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.121 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:02:09.829 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:02:09.829 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:02:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:02:09.829 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:02:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:02:09.829 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:02:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:02:09.829 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:02:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:02:09.829 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:02:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:02:09.829 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:02:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:02:09.829 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:02:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:02:09.829 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:02:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:02:09.829 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:02:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:02:09.829 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:02:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:02:09.829 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:02:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:02:09.829 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:02:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:02:09.829 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:02:09.829 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:02:09.829 139328 ERROR oslo_service.periodic_task >2018-06-25 02:02:39.701 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:02:39.702 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:02:39.718 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:02:39.719 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:02:39.753 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:02:39.753 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:02:39.753 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:02:39.753 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:02:39.753 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:02:39.753 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:02:39.753 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:02:39.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:02:39.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:02:39.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:02:49.756 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:02:49.793 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:02:49.793 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:02:49.793 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:02:49.793 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:02:49.793 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:02:49.793 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:02:49.793 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:02:49.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:02:49.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:02:49.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:03:09.796 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:03:09.828 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:03:09.828 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:03:09.828 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:03:09.828 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:03:09.828 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:03:09.828 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:03:09.828 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:03:09.830 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:03:09.830 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:03:09.830 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:03:09.830 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:03:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:03:09.830 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:03:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:03:09.830 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:03:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:03:09.830 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:03:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:03:09.830 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:03:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:03:09.830 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:03:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:03:09.830 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:03:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:03:09.830 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:03:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:03:09.830 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:03:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:03:09.830 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:03:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:03:09.830 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:03:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:03:09.830 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:03:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:03:09.830 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:03:09.830 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:03:09.830 139328 ERROR oslo_service.periodic_task >2018-06-25 02:03:39.702 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:03:39.703 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:03:39.710 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:03:39.710 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:03:39.744 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:03:39.744 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:03:39.744 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:03:39.744 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:03:39.744 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:03:39.744 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:03:39.744 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:03:39.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:03:39.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:03:39.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:03:49.747 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:03:49.793 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:03:49.793 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:03:49.793 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:03:49.793 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:03:49.793 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:03:49.793 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:03:49.793 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:03:49.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:03:49.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.085 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:03:49.797 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:04:09.797 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:04:09.831 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:04:09.831 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:04:09.831 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:04:09.831 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:04:09.831 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:04:09.831 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:04:09.831 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:04:09.833 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:04:09.833 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.123 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:04:09.834 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:04:09.834 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:04:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:04:09.834 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:04:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:04:09.834 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:04:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:04:09.834 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:04:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:04:09.834 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:04:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:04:09.834 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:04:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:04:09.834 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:04:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:04:09.834 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:04:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:04:09.834 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:04:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:04:09.834 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:04:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:04:09.834 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:04:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:04:09.834 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:04:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:04:09.834 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:04:09.834 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:04:09.834 139328 ERROR oslo_service.periodic_task >2018-06-25 02:04:39.705 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:04:39.706 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:04:39.713 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:04:39.713 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:04:39.745 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:04:39.745 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:04:39.745 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:04:39.745 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:04:39.745 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:04:39.745 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:04:39.745 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:04:39.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:04:39.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:04:39.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:04:49.749 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:04:49.782 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:04:49.782 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:04:49.782 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:04:49.782 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:04:49.782 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:04:49.782 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:04:49.782 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:04:49.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:04:49.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.07 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:04:49.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:05:09.785 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:05:09.819 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:05:09.819 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:05:09.819 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:05:09.819 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:05:09.819 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:05:09.819 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:05:09.819 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:05:09.821 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:05:09.821 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.107 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:05:09.822 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:05:09.822 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:05:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:05:09.822 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:05:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:05:09.822 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:05:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:05:09.822 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:05:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:05:09.822 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:05:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:05:09.822 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:05:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:05:09.822 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:05:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:05:09.822 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:05:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:05:09.822 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:05:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:05:09.822 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:05:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:05:09.822 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:05:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:05:09.822 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:05:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:05:09.822 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:05:09.822 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:05:09.822 139328 ERROR oslo_service.periodic_task >2018-06-25 02:05:39.703 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:05:39.704 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:05:39.711 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:05:39.711 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:05:39.741 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:05:39.741 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:05:39.741 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:05:39.741 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:05:39.741 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:05:39.741 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:05:39.741 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:05:39.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:05:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.031 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:05:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:05:49.744 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:05:49.779 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:05:49.779 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:05:49.779 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:05:49.779 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:05:49.779 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:05:49.779 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:05:49.779 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:05:49.781 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:05:49.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.07 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:05:49.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:06:09.782 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:06:09.814 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:06:09.814 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:06:09.814 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:06:09.814 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:06:09.814 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:06:09.814 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:06:09.814 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:06:09.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:06:09.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.104 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:06:09.817 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:06:09.817 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:06:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:06:09.817 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:06:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:06:09.817 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:06:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:06:09.817 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:06:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:06:09.817 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:06:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:06:09.817 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:06:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:06:09.817 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:06:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:06:09.817 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:06:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:06:09.817 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:06:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:06:09.817 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:06:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:06:09.817 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:06:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:06:09.817 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:06:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:06:09.817 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:06:09.817 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:06:09.817 139328 ERROR oslo_service.periodic_task >2018-06-25 02:06:39.708 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:06:39.709 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:06:39.718 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:06:39.718 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:06:39.752 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:06:39.752 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:06:39.752 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:06:39.752 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:06:39.752 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:06:39.752 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:06:39.752 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:06:39.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:06:39.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:06:39.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:06:49.756 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:06:49.793 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:06:49.793 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:06:49.793 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:06:49.793 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:06:49.793 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:06:49.793 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:06:49.793 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:06:49.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:06:49.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:06:49.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:07:09.797 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:07:09.836 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:07:09.836 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:07:09.836 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:07:09.836 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:07:09.836 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:07:09.836 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:07:09.836 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:07:09.839 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:07:09.839 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.12 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:07:09.839 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:07:09.839 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:07:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:07:09.839 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:07:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:07:09.839 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:07:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:07:09.839 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:07:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:07:09.839 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:07:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:07:09.839 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:07:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:07:09.839 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:07:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:07:09.839 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:07:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:07:09.839 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:07:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:07:09.839 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:07:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:07:09.839 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:07:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:07:09.839 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:07:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:07:09.839 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:07:09.839 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:07:09.839 139328 ERROR oslo_service.periodic_task >2018-06-25 02:07:39.711 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:07:39.712 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:07:39.720 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:07:39.720 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:07:39.755 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:07:39.755 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:07:39.755 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:07:39.755 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:07:39.755 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:07:39.755 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:07:39.755 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:07:39.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:07:39.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:07:39.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:07:49.758 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:07:49.793 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:07:49.793 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:07:49.793 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:07:49.793 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:07:49.793 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:07:49.793 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:07:49.793 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:07:49.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:07:49.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:07:49.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:08:09.796 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:08:09.828 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:08:09.828 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:08:09.828 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:08:09.828 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:08:09.828 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:08:09.828 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:08:09.828 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:08:09.830 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:08:09.830 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:08:09.831 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:08:09.831 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:08:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:08:09.831 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:08:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:08:09.831 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:08:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:08:09.831 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:08:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:08:09.831 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:08:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:08:09.831 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:08:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:08:09.831 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:08:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:08:09.831 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:08:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:08:09.831 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:08:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:08:09.831 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:08:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:08:09.831 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:08:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:08:09.831 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:08:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:08:09.831 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:08:09.831 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:08:09.831 139328 ERROR oslo_service.periodic_task >2018-06-25 02:08:39.712 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:08:39.713 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:08:39.719 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:08:39.720 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:08:39.754 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:08:39.754 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:08:39.754 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:08:39.754 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:08:39.754 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:08:39.754 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:08:39.754 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:08:39.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:08:39.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:08:39.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:08:49.757 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:08:49.789 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:08:49.789 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:08:49.789 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:08:49.789 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:08:49.789 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:08:49.789 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:08:49.789 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:08:49.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:08:49.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:08:49.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:09:09.792 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:09:09.826 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:09:09.826 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:09:09.826 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:09:09.826 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:09:09.826 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:09:09.826 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:09:09.826 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:09:09.828 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:09:09.828 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:09:09.828 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:09:09.828 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:09:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:09:09.828 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:09:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:09:09.828 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:09:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:09:09.828 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:09:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:09:09.828 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:09:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:09:09.828 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:09:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:09:09.828 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:09:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:09:09.828 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:09:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:09:09.828 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:09:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:09:09.828 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:09:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:09:09.828 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:09:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:09:09.828 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:09:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:09:09.828 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:09:09.828 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:09:09.828 139328 ERROR oslo_service.periodic_task >2018-06-25 02:09:39.710 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:09:39.711 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:09:39.718 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:09:39.719 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:09:39.752 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:09:39.752 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:09:39.752 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:09:39.752 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:09:39.752 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:09:39.752 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:09:39.752 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:09:39.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:09:39.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:09:39.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:09:49.755 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:09:49.789 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:09:49.789 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:09:49.789 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:09:49.789 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:09:49.789 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:09:49.789 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:09:49.789 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:09:49.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:09:49.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:09:49.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:10:09.792 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:10:09.832 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:10:09.832 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:10:09.832 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:10:09.832 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:10:09.832 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:10:09.832 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:10:09.832 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:10:09.835 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:10:09.835 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:10:09.836 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:10:09.836 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:10:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:10:09.836 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:10:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:10:09.836 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:10:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:10:09.836 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:10:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:10:09.836 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:10:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:10:09.836 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:10:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:10:09.836 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:10:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:10:09.836 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:10:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:10:09.836 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:10:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:10:09.836 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:10:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:10:09.836 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:10:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:10:09.836 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:10:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:10:09.836 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:10:09.836 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:10:09.836 139328 ERROR oslo_service.periodic_task >2018-06-25 02:10:39.707 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:10:39.708 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:10:39.715 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:10:39.715 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:10:39.748 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:10:39.748 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:10:39.748 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:10:39.748 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:10:39.748 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:10:39.748 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:10:39.748 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:10:39.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:10:39.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:10:39.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:10:49.752 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:10:49.790 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:10:49.790 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:10:49.790 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:10:49.790 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:10:49.790 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:10:49.790 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:10:49.790 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:10:49.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:10:49.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:10:49.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:11:09.793 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:11:09.830 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:11:09.830 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:11:09.830 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:11:09.830 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:11:09.830 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:11:09.830 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:11:09.830 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:11:09.832 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:11:09.833 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:11:09.833 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:11:09.833 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:11:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:11:09.833 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:11:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:11:09.833 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:11:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:11:09.833 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:11:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:11:09.833 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:11:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:11:09.833 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:11:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:11:09.833 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:11:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:11:09.833 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:11:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:11:09.833 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:11:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:11:09.833 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:11:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:11:09.833 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:11:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:11:09.833 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:11:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:11:09.833 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:11:09.833 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:11:09.833 139328 ERROR oslo_service.periodic_task >2018-06-25 02:11:39.705 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:11:39.706 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:11:39.714 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:11:39.715 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:11:39.756 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:11:39.756 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:11:39.756 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:11:39.756 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:11:39.756 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:11:39.756 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:11:39.756 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:11:39.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:11:39.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.043 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:11:39.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:11:49.759 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:11:49.799 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:11:49.799 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:11:49.799 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:11:49.799 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:11:49.799 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:11:49.799 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:11:49.799 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:11:49.801 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:11:49.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.087 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:11:49.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:12:09.802 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:12:09.835 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:12:09.835 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:12:09.835 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:12:09.835 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:12:09.835 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:12:09.835 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:12:09.835 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:12:09.837 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:12:09.838 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.123 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:12:09.838 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:12:09.838 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:12:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:12:09.838 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:12:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:12:09.838 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:12:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:12:09.838 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:12:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:12:09.838 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:12:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:12:09.838 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:12:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:12:09.838 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:12:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:12:09.838 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:12:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:12:09.838 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:12:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:12:09.838 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:12:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:12:09.838 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:12:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:12:09.838 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:12:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:12:09.838 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:12:09.838 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:12:09.838 139328 ERROR oslo_service.periodic_task >2018-06-25 02:12:39.710 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:12:39.711 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:12:39.717 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:12:39.718 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:12:39.751 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:12:39.751 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:12:39.751 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:12:39.751 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:12:39.751 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:12:39.751 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:12:39.751 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:12:39.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:12:39.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:12:39.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:12:49.756 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:12:49.791 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:12:49.791 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:12:49.791 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:12:49.791 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:12:49.791 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:12:49.791 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:12:49.791 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:12:49.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:12:49.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:12:49.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:13:09.794 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:13:09.830 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:13:09.830 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:13:09.830 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:13:09.830 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:13:09.830 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:13:09.830 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:13:09.830 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:13:09.833 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:13:09.833 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:13:09.833 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:13:09.833 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:13:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:13:09.833 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:13:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:13:09.833 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:13:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:13:09.833 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:13:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:13:09.833 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:13:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:13:09.833 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:13:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:13:09.833 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:13:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:13:09.833 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:13:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:13:09.833 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:13:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:13:09.833 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:13:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:13:09.833 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:13:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:13:09.833 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:13:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:13:09.833 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:13:09.833 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:13:09.833 139328 ERROR oslo_service.periodic_task >2018-06-25 02:13:39.705 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:13:39.706 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:13:39.713 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:13:39.714 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:13:39.746 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:13:39.746 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:13:39.746 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:13:39.746 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:13:39.746 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:13:39.746 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:13:39.746 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:13:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:13:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:13:39.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:13:49.751 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:13:49.785 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:13:49.785 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:13:49.785 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:13:49.785 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:13:49.785 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:13:49.785 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:13:49.785 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:13:49.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:13:49.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:13:49.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:14:09.788 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:14:09.823 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:14:09.823 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:14:09.823 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:14:09.823 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:14:09.823 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:14:09.823 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:14:09.823 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:14:09.825 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:14:09.825 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:14:09.825 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:14:09.825 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:14:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:14:09.825 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:14:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:14:09.825 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:14:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:14:09.825 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:14:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:14:09.825 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:14:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:14:09.825 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:14:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:14:09.825 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:14:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:14:09.825 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:14:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:14:09.825 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:14:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:14:09.825 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:14:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:14:09.825 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:14:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:14:09.825 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:14:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:14:09.825 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:14:09.825 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:14:09.825 139328 ERROR oslo_service.periodic_task >2018-06-25 02:14:39.707 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:14:39.707 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:14:39.714 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:14:39.715 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:14:39.748 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:14:39.748 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:14:39.748 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:14:39.748 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:14:39.748 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:14:39.748 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:14:39.748 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:14:39.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:14:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:14:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:14:49.751 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:14:49.783 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:14:49.783 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:14:49.783 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:14:49.783 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:14:49.783 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:14:49.783 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:14:49.783 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:14:49.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:14:49.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.07 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:14:49.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:15:09.786 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:15:09.821 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:15:09.821 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:15:09.821 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:15:09.821 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:15:09.821 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:15:09.821 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:15:09.821 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:15:09.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:15:09.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.107 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:15:09.824 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:15:09.824 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:15:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:15:09.824 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:15:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:15:09.824 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:15:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:15:09.824 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:15:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:15:09.824 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:15:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:15:09.824 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:15:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:15:09.824 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:15:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:15:09.824 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:15:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:15:09.824 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:15:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:15:09.824 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:15:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:15:09.824 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:15:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:15:09.824 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:15:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:15:09.824 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:15:09.824 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:15:09.824 139328 ERROR oslo_service.periodic_task >2018-06-25 02:15:39.706 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:15:39.707 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:15:39.716 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:15:39.717 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:15:39.751 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:15:39.751 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:15:39.751 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:15:39.751 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:15:39.751 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:15:39.751 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:15:39.751 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:15:39.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:15:39.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:15:39.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:15:49.754 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:15:49.786 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:15:49.786 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:15:49.786 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:15:49.786 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:15:49.786 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:15:49.786 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:15:49.786 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:15:49.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:15:49.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:15:49.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:16:09.789 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:16:09.823 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:16:09.823 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:16:09.823 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:16:09.823 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:16:09.823 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:16:09.823 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:16:09.823 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:16:09.825 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:16:09.825 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:16:09.825 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:16:09.825 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:16:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:16:09.825 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:16:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:16:09.825 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:16:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:16:09.825 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:16:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:16:09.825 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:16:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:16:09.825 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:16:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:16:09.825 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:16:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:16:09.825 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:16:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:16:09.825 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:16:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:16:09.825 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:16:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:16:09.825 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:16:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:16:09.825 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:16:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:16:09.825 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:16:09.825 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:16:09.825 139328 ERROR oslo_service.periodic_task >2018-06-25 02:16:39.707 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:16:39.709 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:16:39.716 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:16:39.716 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:16:39.753 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:16:39.753 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:16:39.753 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:16:39.753 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:16:39.753 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:16:39.753 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:16:39.753 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:16:39.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:16:39.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:16:39.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:16:49.757 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:16:49.789 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:16:49.789 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:16:49.789 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:16:49.789 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:16:49.789 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:16:49.789 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:16:49.789 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:16:49.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:16:49.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:16:49.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:17:09.792 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:17:09.834 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:17:09.834 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:17:09.834 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:17:09.834 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:17:09.834 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:17:09.834 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:17:09.834 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:17:09.836 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:17:09.836 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.12 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:17:09.837 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:17:09.837 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:17:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:17:09.837 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:17:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:17:09.837 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:17:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:17:09.837 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:17:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:17:09.837 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:17:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:17:09.837 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:17:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:17:09.837 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:17:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:17:09.837 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:17:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:17:09.837 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:17:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:17:09.837 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:17:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:17:09.837 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:17:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:17:09.837 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:17:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:17:09.837 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:17:09.837 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:17:09.837 139328 ERROR oslo_service.periodic_task >2018-06-25 02:17:39.708 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:17:39.709 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:17:39.716 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:17:39.717 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:17:39.751 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:17:39.751 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:17:39.751 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:17:39.751 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:17:39.751 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:17:39.751 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:17:39.751 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:17:39.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:17:39.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:17:39.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:17:49.755 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:17:49.787 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:17:49.787 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:17:49.787 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:17:49.787 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:17:49.787 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:17:49.787 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:17:49.787 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:17:49.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:17:49.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:17:49.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:18:09.791 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:18:09.823 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:18:09.823 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:18:09.823 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:18:09.823 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:18:09.823 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:18:09.823 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:18:09.823 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:18:09.825 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:18:09.826 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:18:09.826 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:18:09.826 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:18:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:18:09.826 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:18:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:18:09.826 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:18:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:18:09.826 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:18:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:18:09.826 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:18:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:18:09.826 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:18:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:18:09.826 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:18:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:18:09.826 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:18:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:18:09.826 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:18:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:18:09.826 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:18:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:18:09.826 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:18:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:18:09.826 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:18:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:18:09.826 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:18:09.826 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:18:09.826 139328 ERROR oslo_service.periodic_task >2018-06-25 02:18:39.707 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:18:39.709 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:18:39.716 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:18:39.717 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:18:39.753 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:18:39.753 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:18:39.753 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:18:39.753 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:18:39.753 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:18:39.753 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:18:39.753 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:18:39.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:18:39.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:18:39.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:18:49.756 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:18:49.789 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:18:49.789 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:18:49.789 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:18:49.789 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:18:49.789 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:18:49.789 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:18:49.789 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:18:49.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:18:49.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:18:49.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:19:09.792 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:19:09.826 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:19:09.826 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:19:09.826 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:19:09.826 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:19:09.826 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:19:09.826 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:19:09.826 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:19:09.829 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:19:09.829 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:19:09.830 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:19:09.830 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:19:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:19:09.830 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:19:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:19:09.830 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:19:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:19:09.830 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:19:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:19:09.830 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:19:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:19:09.830 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:19:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:19:09.830 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:19:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:19:09.830 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:19:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:19:09.830 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:19:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:19:09.830 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:19:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:19:09.830 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:19:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:19:09.830 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:19:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:19:09.830 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:19:09.830 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:19:09.830 139328 ERROR oslo_service.periodic_task >2018-06-25 02:19:39.712 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:19:39.712 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:19:39.720 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:19:39.721 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:19:39.755 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:19:39.755 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:19:39.755 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:19:39.755 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:19:39.755 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:19:39.755 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:19:39.755 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:19:39.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:19:39.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:19:39.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:19:49.759 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:19:49.791 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:19:49.791 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:19:49.791 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:19:49.791 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:19:49.791 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:19:49.791 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:19:49.791 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:19:49.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:19:49.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:19:49.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:20:09.794 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:20:09.825 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:20:09.825 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:20:09.825 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:20:09.825 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:20:09.825 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:20:09.825 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:20:09.825 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:20:09.828 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:20:09.828 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.106 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:20:09.828 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:20:09.828 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:20:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:20:09.828 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:20:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:20:09.828 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:20:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:20:09.828 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:20:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:20:09.828 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:20:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:20:09.828 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:20:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:20:09.828 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:20:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:20:09.828 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:20:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:20:09.828 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:20:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:20:09.828 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:20:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:20:09.828 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:20:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:20:09.828 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:20:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:20:09.828 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:20:09.828 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:20:09.828 139328 ERROR oslo_service.periodic_task >2018-06-25 02:20:39.711 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:20:39.713 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:20:39.720 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:20:39.720 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:20:39.756 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:20:39.756 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:20:39.756 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:20:39.756 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:20:39.756 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:20:39.756 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:20:39.756 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:20:39.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:20:39.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:20:39.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:20:49.759 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:20:49.799 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:20:49.799 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:20:49.799 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:20:49.799 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:20:49.799 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:20:49.799 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:20:49.799 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:20:49.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:20:49.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.081 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:20:49.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:21:09.803 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:21:09.836 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:21:09.836 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:21:09.836 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:21:09.836 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:21:09.836 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:21:09.836 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:21:09.836 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:21:09.839 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:21:09.839 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.118 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:21:09.840 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:21:09.840 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:21:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:21:09.840 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:21:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:21:09.840 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:21:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:21:09.840 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:21:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:21:09.840 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:21:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:21:09.840 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:21:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:21:09.840 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:21:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:21:09.840 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:21:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:21:09.840 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:21:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:21:09.840 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:21:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:21:09.840 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:21:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:21:09.840 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:21:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:21:09.840 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:21:09.840 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:21:09.840 139328 ERROR oslo_service.periodic_task >2018-06-25 02:21:39.712 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:21:39.712 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:21:39.721 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:21:39.721 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:21:39.754 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:21:39.754 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:21:39.754 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:21:39.754 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:21:39.754 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:21:39.754 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:21:39.754 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:21:39.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:21:39.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:21:39.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:21:49.757 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:21:49.790 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:21:49.790 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:21:49.790 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:21:49.790 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:21:49.790 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:21:49.790 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:21:49.790 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:21:49.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:21:49.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:21:49.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:22:09.793 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:22:09.826 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:22:09.826 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:22:09.826 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:22:09.826 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:22:09.826 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:22:09.826 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:22:09.826 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:22:09.828 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:22:09.828 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.106 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:22:09.829 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:22:09.829 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:22:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:22:09.829 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:22:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:22:09.829 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:22:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:22:09.829 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:22:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:22:09.829 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:22:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:22:09.829 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:22:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:22:09.829 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:22:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:22:09.829 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:22:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:22:09.829 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:22:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:22:09.829 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:22:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:22:09.829 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:22:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:22:09.829 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:22:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:22:09.829 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:22:09.829 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:22:09.829 139328 ERROR oslo_service.periodic_task >2018-06-25 02:22:39.711 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:22:39.712 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:22:39.724 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:22:39.725 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:22:39.760 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:22:39.760 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:22:39.760 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:22:39.760 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:22:39.760 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:22:39.760 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:22:39.760 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:22:39.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:22:39.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:22:39.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:22:49.763 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:22:49.796 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:22:49.796 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:22:49.796 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:22:49.796 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:22:49.796 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:22:49.796 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:22:49.796 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:22:49.799 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:22:49.799 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:22:49.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:23:09.800 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:23:09.831 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:23:09.831 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:23:09.831 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:23:09.831 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:23:09.831 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:23:09.831 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:23:09.831 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:23:09.833 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:23:09.834 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:23:09.834 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:23:09.834 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:23:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:23:09.834 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:23:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:23:09.834 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:23:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:23:09.834 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:23:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:23:09.834 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:23:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:23:09.834 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:23:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:23:09.834 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:23:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:23:09.834 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:23:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:23:09.834 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:23:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:23:09.834 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:23:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:23:09.834 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:23:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:23:09.834 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:23:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:23:09.834 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:23:09.834 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:23:09.834 139328 ERROR oslo_service.periodic_task >2018-06-25 02:23:39.716 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:23:39.716 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:23:39.723 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:23:39.723 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:23:39.759 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:23:39.759 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:23:39.759 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:23:39.759 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:23:39.759 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:23:39.759 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:23:39.759 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:23:39.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:23:39.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:23:39.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:23:49.762 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:23:49.796 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:23:49.796 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:23:49.796 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:23:49.796 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:23:49.796 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:23:49.796 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:23:49.796 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:23:49.798 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:23:49.799 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:23:49.799 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:24:09.799 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:24:09.836 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:24:09.836 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:24:09.836 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:24:09.836 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:24:09.836 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:24:09.836 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:24:09.836 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:24:09.839 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:24:09.839 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:24:09.840 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:24:09.840 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:24:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:24:09.840 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:24:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:24:09.840 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:24:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:24:09.840 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:24:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:24:09.840 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:24:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:24:09.840 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:24:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:24:09.840 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:24:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:24:09.840 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:24:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:24:09.840 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:24:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:24:09.840 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:24:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:24:09.840 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:24:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:24:09.840 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:24:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:24:09.840 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:24:09.840 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:24:09.840 139328 ERROR oslo_service.periodic_task >2018-06-25 02:24:39.711 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:24:39.713 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:24:39.721 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:24:39.721 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:24:39.755 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:24:39.755 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:24:39.755 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:24:39.755 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:24:39.755 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:24:39.755 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:24:39.755 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:24:39.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:24:39.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:24:39.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:24:49.759 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:24:49.794 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:24:49.794 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:24:49.794 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:24:49.794 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:24:49.794 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:24:49.794 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:24:49.794 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:24:49.797 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:24:49.797 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:24:49.798 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:25:09.798 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:25:09.833 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:25:09.833 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:25:09.833 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:25:09.833 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:25:09.833 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:25:09.833 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:25:09.833 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:25:09.835 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:25:09.835 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:25:09.835 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:25:09.835 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:25:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:25:09.835 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:25:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:25:09.835 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:25:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:25:09.835 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:25:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:25:09.835 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:25:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:25:09.835 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:25:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:25:09.835 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:25:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:25:09.835 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:25:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:25:09.835 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:25:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:25:09.835 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:25:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:25:09.835 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:25:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:25:09.835 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:25:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:25:09.835 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:25:09.835 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:25:09.835 139328 ERROR oslo_service.periodic_task >2018-06-25 02:25:39.707 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:25:39.708 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:25:39.714 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:25:39.715 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:25:39.747 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:25:39.747 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:25:39.747 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:25:39.747 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:25:39.747 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:25:39.747 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:25:39.747 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:25:39.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:25:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:25:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:25:49.750 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:25:49.783 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:25:49.783 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:25:49.783 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:25:49.783 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:25:49.783 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:25:49.783 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:25:49.783 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:25:49.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:25:49.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.07 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:25:49.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:26:09.786 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:26:09.820 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:26:09.820 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:26:09.820 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:26:09.820 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:26:09.820 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:26:09.820 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:26:09.820 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:26:09.822 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:26:09.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:26:09.823 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:26:09.823 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:26:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:26:09.823 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:26:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:26:09.823 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:26:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:26:09.823 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:26:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:26:09.823 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:26:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:26:09.823 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:26:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:26:09.823 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:26:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:26:09.823 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:26:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:26:09.823 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:26:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:26:09.823 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:26:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:26:09.823 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:26:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:26:09.823 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:26:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:26:09.823 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:26:09.823 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:26:09.823 139328 ERROR oslo_service.periodic_task >2018-06-25 02:26:39.705 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:26:39.706 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:26:39.712 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:26:39.713 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:26:39.759 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:26:39.759 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:26:39.759 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:26:39.759 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:26:39.759 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:26:39.759 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:26:39.759 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:26:39.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:26:39.764 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.051 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:26:39.764 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:26:49.765 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:26:49.799 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:26:49.799 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:26:49.799 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:26:49.799 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:26:49.799 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:26:49.799 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:26:49.799 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:26:49.801 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:26:49.801 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.088 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:26:49.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:27:09.802 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:27:09.836 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:27:09.836 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:27:09.836 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:27:09.836 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:27:09.836 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:27:09.836 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:27:09.836 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:27:09.838 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:27:09.838 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.126 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:27:09.839 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:27:09.839 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:27:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:27:09.839 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:27:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:27:09.839 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:27:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:27:09.839 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:27:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:27:09.839 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:27:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:27:09.839 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:27:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:27:09.839 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:27:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:27:09.839 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:27:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:27:09.839 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:27:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:27:09.839 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:27:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:27:09.839 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:27:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:27:09.839 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:27:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:27:09.839 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:27:09.839 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:27:09.839 139328 ERROR oslo_service.periodic_task >2018-06-25 02:27:39.700 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:27:39.701 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:27:39.709 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:27:39.709 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:27:39.746 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:27:39.746 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:27:39.746 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:27:39.746 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:27:39.746 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:27:39.746 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:27:39.746 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:27:39.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:27:39.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:27:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:27:49.750 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:27:49.785 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:27:49.785 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:27:49.785 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:27:49.785 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:27:49.785 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:27:49.785 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:27:49.785 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:27:49.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:27:49.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:27:49.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:28:09.788 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:28:09.821 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:28:09.821 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:28:09.821 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:28:09.821 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:28:09.821 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:28:09.821 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:28:09.821 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:28:09.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:28:09.824 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:28:09.824 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:28:09.824 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:28:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:28:09.824 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:28:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:28:09.824 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:28:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:28:09.824 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:28:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:28:09.824 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:28:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:28:09.824 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:28:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:28:09.824 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:28:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:28:09.824 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:28:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:28:09.824 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:28:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:28:09.824 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:28:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:28:09.824 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:28:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:28:09.824 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:28:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:28:09.824 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:28:09.824 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:28:09.824 139328 ERROR oslo_service.periodic_task >2018-06-25 02:28:39.695 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:28:39.696 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:28:39.707 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:28:39.707 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:28:39.742 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:28:39.742 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:28:39.742 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:28:39.742 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:28:39.742 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:28:39.742 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:28:39.742 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:28:39.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:28:39.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:28:39.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:28:49.745 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:28:49.778 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:28:49.778 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:28:49.778 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:28:49.778 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:28:49.778 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:28:49.778 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:28:49.778 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:28:49.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:28:49.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:28:49.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:29:09.781 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:29:09.814 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:29:09.814 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:29:09.814 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:29:09.814 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:29:09.814 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:29:09.814 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:29:09.814 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:29:09.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:29:09.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:29:09.816 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:29:09.816 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:29:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:29:09.816 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:29:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:29:09.816 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:29:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:29:09.816 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:29:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:29:09.816 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:29:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:29:09.816 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:29:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:29:09.816 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:29:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:29:09.816 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:29:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:29:09.816 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:29:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:29:09.816 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:29:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:29:09.816 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:29:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:29:09.816 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:29:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:29:09.816 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:29:09.816 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:29:09.816 139328 ERROR oslo_service.periodic_task >2018-06-25 02:29:39.698 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:29:39.699 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:29:39.706 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:29:39.706 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:29:39.739 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:29:39.739 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:29:39.739 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:29:39.739 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:29:39.739 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:29:39.739 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:29:39.739 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:29:39.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:29:39.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:29:39.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:29:49.742 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:29:49.776 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:29:49.776 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:29:49.776 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:29:49.776 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:29:49.776 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:29:49.776 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:29:49.776 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:29:49.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:29:49.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:29:49.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:30:09.780 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:30:09.813 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:30:09.813 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:30:09.813 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:30:09.813 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:30:09.813 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:30:09.813 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:30:09.813 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:30:09.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:30:09.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:30:09.818 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:30:09.818 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:30:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:30:09.818 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:30:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:30:09.818 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:30:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:30:09.818 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:30:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:30:09.818 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:30:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:30:09.818 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:30:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:30:09.818 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:30:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:30:09.818 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:30:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:30:09.818 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:30:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:30:09.818 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:30:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:30:09.818 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:30:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:30:09.818 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:30:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:30:09.818 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:30:09.818 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:30:09.818 139328 ERROR oslo_service.periodic_task >2018-06-25 02:30:39.700 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:30:39.701 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:30:39.708 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:30:39.709 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:30:39.742 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:30:39.742 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:30:39.742 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:30:39.742 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:30:39.742 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:30:39.742 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:30:39.742 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:30:39.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:30:39.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:30:39.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:30:49.746 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:30:49.777 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:30:49.777 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:30:49.777 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:30:49.777 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:30:49.777 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:30:49.777 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:30:49.777 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:30:49.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:30:49.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.07 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:30:49.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:31:09.780 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:31:09.814 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:31:09.814 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:31:09.814 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:31:09.814 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:31:09.814 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:31:09.814 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:31:09.814 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:31:09.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:31:09.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.107 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:31:09.817 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:31:09.817 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:31:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:31:09.817 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:31:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:31:09.817 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:31:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:31:09.817 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:31:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:31:09.817 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:31:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:31:09.817 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:31:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:31:09.817 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:31:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:31:09.817 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:31:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:31:09.817 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:31:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:31:09.817 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:31:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:31:09.817 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:31:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:31:09.817 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:31:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:31:09.817 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:31:09.817 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:31:09.817 139328 ERROR oslo_service.periodic_task >2018-06-25 02:31:39.698 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:31:39.699 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:31:39.708 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:31:39.708 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:31:39.749 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:31:39.749 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:31:39.749 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:31:39.749 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:31:39.749 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:31:39.749 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:31:39.749 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:31:39.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:31:39.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.043 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:31:39.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:31:49.753 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:31:49.786 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:31:49.786 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:31:49.786 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:31:49.786 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:31:49.786 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:31:49.786 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:31:49.786 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:31:49.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:31:49.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:31:49.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:32:09.789 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:32:09.823 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:32:09.823 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:32:09.823 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:32:09.823 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:32:09.823 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:32:09.823 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:32:09.823 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:32:09.826 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:32:09.826 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:32:09.827 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:32:09.827 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:32:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:32:09.827 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:32:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:32:09.827 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:32:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:32:09.827 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:32:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:32:09.827 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:32:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:32:09.827 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:32:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:32:09.827 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:32:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:32:09.827 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:32:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:32:09.827 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:32:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:32:09.827 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:32:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:32:09.827 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:32:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:32:09.827 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:32:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:32:09.827 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:32:09.827 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:32:09.827 139328 ERROR oslo_service.periodic_task >2018-06-25 02:32:39.698 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:32:39.699 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:32:39.707 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:32:39.708 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:32:39.741 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:32:39.741 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:32:39.741 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:32:39.741 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:32:39.741 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:32:39.741 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:32:39.741 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:32:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:32:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:32:39.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:32:49.744 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:32:49.775 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:32:49.775 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:32:49.775 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:32:49.775 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:32:49.775 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:32:49.775 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:32:49.775 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:32:49.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:32:49.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.07 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:32:49.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:33:09.778 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:33:09.812 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:33:09.812 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:33:09.812 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:33:09.812 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:33:09.812 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:33:09.812 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:33:09.812 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:33:09.814 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:33:09.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.107 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:33:09.815 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:33:09.815 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:33:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:33:09.815 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:33:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:33:09.815 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:33:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:33:09.815 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:33:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:33:09.815 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:33:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:33:09.815 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:33:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:33:09.815 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:33:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:33:09.815 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:33:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:33:09.815 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:33:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:33:09.815 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:33:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:33:09.815 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:33:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:33:09.815 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:33:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:33:09.815 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:33:09.815 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:33:09.815 139328 ERROR oslo_service.periodic_task >2018-06-25 02:33:39.697 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:33:39.698 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:33:39.706 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:33:39.706 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:33:39.762 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:33:39.762 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:33:39.762 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:33:39.762 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:33:39.762 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:33:39.762 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:33:39.762 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:33:39.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:33:39.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.058 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:33:39.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:33:49.766 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:33:49.799 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:33:49.799 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:33:49.799 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:33:49.799 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:33:49.799 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:33:49.799 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:33:49.799 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:33:49.801 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:33:49.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.095 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:33:49.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:34:09.802 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:34:09.845 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:34:09.845 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:34:09.845 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:34:09.845 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:34:09.845 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:34:09.845 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:34:09.845 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:34:09.847 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:34:09.848 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.141 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:34:09.848 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:34:09.848 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:34:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:34:09.848 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:34:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:34:09.848 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:34:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:34:09.848 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:34:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:34:09.848 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:34:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:34:09.848 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:34:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:34:09.848 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:34:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:34:09.848 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:34:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:34:09.848 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:34:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:34:09.848 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:34:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:34:09.848 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:34:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:34:09.848 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:34:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:34:09.848 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:34:09.848 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:34:09.848 139328 ERROR oslo_service.periodic_task >2018-06-25 02:34:39.699 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:34:39.700 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:34:39.710 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:34:39.710 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:34:39.743 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:34:39.743 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:34:39.743 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:34:39.743 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:34:39.743 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:34:39.743 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:34:39.743 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:34:39.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:34:39.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:34:39.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:34:49.746 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:34:49.780 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:34:49.780 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:34:49.780 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:34:49.780 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:34:49.780 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:34:49.780 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:34:49.780 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:34:49.783 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:34:49.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:34:49.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:35:09.785 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:35:09.818 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:35:09.818 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:35:09.818 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:35:09.818 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:35:09.818 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:35:09.818 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:35:09.818 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:35:09.820 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:35:09.820 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:35:09.820 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:35:09.820 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:35:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:35:09.820 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:35:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:35:09.820 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:35:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:35:09.820 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:35:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:35:09.820 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:35:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:35:09.820 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:35:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:35:09.820 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:35:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:35:09.820 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:35:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:35:09.820 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:35:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:35:09.820 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:35:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:35:09.820 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:35:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:35:09.820 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:35:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:35:09.820 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:35:09.820 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:35:09.820 139328 ERROR oslo_service.periodic_task >2018-06-25 02:35:39.702 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:35:39.703 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:35:39.717 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:35:39.718 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:35:39.755 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:35:39.755 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:35:39.755 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:35:39.755 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:35:39.755 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:35:39.755 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:35:39.755 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:35:39.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:35:39.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:35:39.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:35:49.759 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:35:49.796 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:35:49.796 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:35:49.796 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:35:49.796 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:35:49.796 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:35:49.796 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:35:49.796 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:35:49.799 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:35:49.799 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.081 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:35:49.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:36:09.801 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:36:09.835 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:36:09.835 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:36:09.835 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:36:09.835 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:36:09.835 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:36:09.835 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:36:09.835 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:36:09.837 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:36:09.838 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.12 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:36:09.838 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:36:09.838 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:36:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:36:09.838 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:36:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:36:09.838 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:36:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:36:09.838 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:36:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:36:09.838 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:36:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:36:09.838 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:36:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:36:09.838 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:36:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:36:09.838 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:36:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:36:09.838 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:36:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:36:09.838 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:36:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:36:09.838 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:36:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:36:09.838 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:36:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:36:09.838 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:36:09.838 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:36:09.838 139328 ERROR oslo_service.periodic_task >2018-06-25 02:36:39.700 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:36:39.701 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:36:39.709 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:36:39.709 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:36:39.762 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:36:39.762 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:36:39.762 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:36:39.762 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:36:39.762 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:36:39.762 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:36:39.762 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:36:39.764 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:36:39.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.055 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:36:39.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:36:49.766 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:36:49.806 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:36:49.806 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:36:49.806 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:36:49.806 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:36:49.806 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:36:49.806 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:36:49.806 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:36:49.808 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:36:49.809 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.099 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:36:49.809 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:37:09.809 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:37:09.843 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:37:09.843 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:37:09.843 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:37:09.843 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:37:09.843 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:37:09.843 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:37:09.843 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:37:09.846 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:37:09.846 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.136 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:37:09.846 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:37:09.846 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:37:09.846 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:37:09.846 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:37:09.846 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:37:09.846 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:37:09.846 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:37:09.846 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:37:09.846 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:37:09.846 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:37:09.846 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:37:09.846 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:37:09.846 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:37:09.846 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:37:09.846 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:37:09.846 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:37:09.846 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:37:09.846 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:37:09.846 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:37:09.846 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:37:09.846 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:37:09.846 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:37:09.846 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:37:09.846 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:37:09.846 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:37:09.846 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:37:09.846 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:37:09.846 139328 ERROR oslo_service.periodic_task >2018-06-25 02:37:39.698 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:37:39.699 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:37:39.705 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:37:39.706 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:37:39.738 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:37:39.738 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:37:39.738 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:37:39.738 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:37:39.738 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:37:39.738 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:37:39.738 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:37:39.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:37:39.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:37:39.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:37:49.741 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:37:49.780 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:37:49.780 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:37:49.780 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:37:49.780 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:37:49.780 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:37:49.780 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:37:49.780 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:37:49.783 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:37:49.783 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:37:49.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:38:09.784 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:38:09.819 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:38:09.819 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:38:09.819 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:38:09.819 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:38:09.819 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:38:09.819 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:38:09.819 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:38:09.821 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:38:09.822 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:38:09.822 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:38:09.822 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:38:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:38:09.822 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:38:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:38:09.822 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:38:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:38:09.822 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:38:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:38:09.822 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:38:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:38:09.822 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:38:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:38:09.822 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:38:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:38:09.822 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:38:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:38:09.822 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:38:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:38:09.822 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:38:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:38:09.822 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:38:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:38:09.822 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:38:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:38:09.822 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:38:09.822 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:38:09.822 139328 ERROR oslo_service.periodic_task >2018-06-25 02:38:39.694 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:38:39.695 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:38:39.701 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:38:39.702 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:38:39.738 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:38:39.738 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:38:39.738 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:38:39.738 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:38:39.738 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:38:39.738 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:38:39.738 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:38:39.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:38:39.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:38:39.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:38:49.742 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:38:49.775 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:38:49.775 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:38:49.775 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:38:49.775 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:38:49.775 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:38:49.775 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:38:49.775 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:38:49.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:38:49.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:38:49.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:39:09.778 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:39:09.816 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:39:09.816 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:39:09.816 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:39:09.816 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:39:09.816 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:39:09.816 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:39:09.816 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:39:09.818 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:39:09.818 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:39:09.819 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:39:09.819 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:39:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:39:09.819 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:39:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:39:09.819 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:39:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:39:09.819 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:39:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:39:09.819 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:39:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:39:09.819 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:39:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:39:09.819 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:39:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:39:09.819 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:39:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:39:09.819 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:39:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:39:09.819 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:39:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:39:09.819 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:39:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:39:09.819 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:39:09.819 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:39:09.819 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:39:09.819 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:39:09.819 139328 ERROR oslo_service.periodic_task >2018-06-25 02:39:39.690 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:39:39.691 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:39:39.699 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:39:39.700 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:39:39.739 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:39:39.739 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:39:39.739 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:39:39.739 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:39:39.739 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:39:39.739 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:39:39.739 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:39:39.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:39:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.042 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:39:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:39:49.744 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:39:49.776 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:39:49.776 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:39:49.776 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:39:49.776 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:39:49.776 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:39:49.776 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:39:49.776 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:39:49.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:39:49.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:39:49.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:40:09.779 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:40:09.812 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:40:09.812 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:40:09.812 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:40:09.812 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:40:09.812 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:40:09.812 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:40:09.812 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:40:09.814 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:40:09.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:40:09.815 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:40:09.815 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:40:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:40:09.815 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:40:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:40:09.815 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:40:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:40:09.815 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:40:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:40:09.815 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:40:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:40:09.815 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:40:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:40:09.815 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:40:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:40:09.815 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:40:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:40:09.815 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:40:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:40:09.815 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:40:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:40:09.815 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:40:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:40:09.815 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:40:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:40:09.815 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:40:09.815 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:40:09.815 139328 ERROR oslo_service.periodic_task >2018-06-25 02:40:39.686 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:40:39.688 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:40:39.694 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:40:39.695 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:40:39.730 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:40:39.730 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:40:39.730 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:40:39.730 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:40:39.730 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:40:39.730 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:40:39.730 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:40:39.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:40:39.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:40:39.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:40:49.734 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:40:49.774 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:40:49.774 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:40:49.774 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:40:49.774 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:40:49.774 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:40:49.774 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:40:49.774 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:40:49.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:40:49.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.083 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:40:49.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:41:09.779 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:41:09.812 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:41:09.812 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:41:09.812 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:41:09.812 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:41:09.812 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:41:09.812 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:41:09.812 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:41:09.814 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:41:09.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.12 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:41:09.815 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:41:09.815 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:41:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:41:09.815 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:41:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:41:09.815 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:41:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:41:09.815 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:41:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:41:09.815 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:41:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:41:09.815 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:41:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:41:09.815 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:41:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:41:09.815 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:41:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:41:09.815 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:41:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:41:09.815 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:41:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:41:09.815 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:41:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:41:09.815 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:41:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:41:09.815 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:41:09.815 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:41:09.815 139328 ERROR oslo_service.periodic_task >2018-06-25 02:41:39.686 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:41:39.688 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:42:39.685 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:42:39.686 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:42:39.694 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:42:39.695 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:42:39.730 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:42:39.730 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:42:39.730 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:42:39.730 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:42:39.730 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:42:39.730 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:42:39.730 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:42:39.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:42:39.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:42:39.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:42:49.734 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:42:49.770 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:42:49.770 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:42:49.770 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:42:49.770 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:42:49.770 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:42:49.770 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:42:49.770 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:42:49.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:42:49.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:42:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:43:09.773 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:43:09.805 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:43:09.805 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:43:09.805 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:43:09.805 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:43:09.805 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:43:09.805 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:43:09.805 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:43:09.807 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:43:09.807 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:43:09.808 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:43:09.808 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:43:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:43:09.808 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:43:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:43:09.808 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:43:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:43:09.808 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:43:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:43:09.808 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:43:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:43:09.808 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:43:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:43:09.808 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:43:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:43:09.808 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:43:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:43:09.808 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:43:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:43:09.808 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:43:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:43:09.808 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:43:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:43:09.808 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:43:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:43:09.808 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:43:09.808 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:43:09.808 139328 ERROR oslo_service.periodic_task >2018-06-25 02:43:39.689 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:43:39.689 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:43:39.695 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:43:39.696 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:43:39.728 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:43:39.728 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:43:39.728 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:43:39.728 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:43:39.728 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:43:39.728 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:43:39.728 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:43:39.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:43:39.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:43:39.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:43:49.731 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:43:49.769 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:43:49.769 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:43:49.769 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:43:49.769 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:43:49.769 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:43:49.769 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:43:49.769 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:43:49.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:43:49.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:43:49.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:44:09.772 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:44:09.804 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:44:09.804 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:44:09.804 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:44:09.804 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:44:09.804 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:44:09.804 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:44:09.804 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:44:09.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:44:09.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:44:09.807 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:44:09.807 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:44:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:44:09.807 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:44:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:44:09.807 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:44:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:44:09.807 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:44:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:44:09.807 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:44:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:44:09.807 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:44:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:44:09.807 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:44:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:44:09.807 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:44:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:44:09.807 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:44:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:44:09.807 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:44:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:44:09.807 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:44:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:44:09.807 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:44:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:44:09.807 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:44:09.807 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:44:09.807 139328 ERROR oslo_service.periodic_task >2018-06-25 02:44:39.688 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:44:39.689 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:44:39.695 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:44:39.696 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:44:39.731 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:44:39.731 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:44:39.731 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:44:39.731 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:44:39.731 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:44:39.731 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:44:39.731 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:44:39.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:44:39.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:44:39.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:44:49.734 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:44:49.770 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:44:49.770 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:44:49.770 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:44:49.770 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:44:49.770 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:44:49.770 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:44:49.770 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:44:49.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:44:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:44:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:45:09.773 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:45:09.809 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:45:09.809 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:45:09.809 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:45:09.809 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:45:09.809 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:45:09.809 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:45:09.809 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:45:09.811 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:45:09.811 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:45:09.811 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:45:09.811 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:45:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:45:09.811 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:45:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:45:09.811 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:45:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:45:09.811 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:45:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:45:09.811 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:45:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:45:09.811 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:45:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:45:09.811 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:45:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:45:09.811 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:45:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:45:09.811 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:45:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:45:09.811 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:45:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:45:09.811 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:45:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:45:09.811 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:45:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:45:09.811 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:45:09.811 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:45:09.811 139328 ERROR oslo_service.periodic_task >2018-06-25 02:45:39.693 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:45:39.694 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:45:39.700 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:45:39.701 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:45:39.732 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:45:39.732 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:45:39.732 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:45:39.732 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:45:39.732 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:45:39.732 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:45:39.732 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:45:39.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:45:39.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:45:39.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:45:49.736 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:45:49.771 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:45:49.771 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:45:49.771 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:45:49.771 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:45:49.771 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:45:49.771 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:45:49.771 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:45:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:45:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:45:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:46:09.774 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:46:09.812 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:46:09.812 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:46:09.812 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:46:09.812 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:46:09.812 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:46:09.812 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:46:09.812 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:46:09.814 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:46:09.814 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:46:09.814 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:46:09.814 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:46:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:46:09.814 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:46:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:46:09.814 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:46:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:46:09.814 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:46:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:46:09.814 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:46:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:46:09.814 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:46:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:46:09.814 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:46:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:46:09.814 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:46:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:46:09.814 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:46:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:46:09.814 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:46:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:46:09.814 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:46:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:46:09.814 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:46:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:46:09.814 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:46:09.814 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:46:09.814 139328 ERROR oslo_service.periodic_task >2018-06-25 02:46:39.696 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:46:39.697 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:46:39.705 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:46:39.706 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:46:39.739 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:46:39.739 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:46:39.739 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:46:39.739 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:46:39.739 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:46:39.739 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:46:39.739 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:46:39.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:46:39.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:46:39.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:46:49.742 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:46:49.779 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:46:49.779 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:46:49.779 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:46:49.779 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:46:49.779 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:46:49.779 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:46:49.779 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:46:49.781 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:46:49.781 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:46:49.781 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:47:09.782 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:47:09.821 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:47:09.821 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:47:09.821 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:47:09.821 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:47:09.821 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:47:09.821 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:47:09.821 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:47:09.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:47:09.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:47:09.823 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:47:09.823 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:47:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:47:09.823 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:47:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:47:09.823 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:47:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:47:09.823 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:47:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:47:09.823 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:47:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:47:09.823 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:47:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:47:09.823 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:47:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:47:09.823 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:47:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:47:09.823 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:47:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:47:09.823 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:47:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:47:09.823 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:47:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:47:09.823 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:47:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:47:09.823 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:47:09.823 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:47:09.823 139328 ERROR oslo_service.periodic_task >2018-06-25 02:47:39.695 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:47:39.696 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:47:39.702 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:47:39.703 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:47:39.740 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:47:39.740 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:47:39.740 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:47:39.740 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:47:39.740 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:47:39.740 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:47:39.740 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:47:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:47:39.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:47:39.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:47:49.745 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:47:49.778 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:47:49.778 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:47:49.778 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:47:49.778 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:47:49.778 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:47:49.778 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:47:49.778 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:47:49.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:47:49.781 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:47:49.781 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:48:09.781 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:48:09.817 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:48:09.817 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:48:09.817 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:48:09.817 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:48:09.817 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:48:09.817 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:48:09.817 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:48:09.819 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:48:09.819 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:48:09.820 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:48:09.820 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:48:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:48:09.820 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:48:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:48:09.820 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:48:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:48:09.820 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:48:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:48:09.820 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:48:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:48:09.820 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:48:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:48:09.820 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:48:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:48:09.820 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:48:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:48:09.820 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:48:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:48:09.820 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:48:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:48:09.820 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:48:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:48:09.820 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:48:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:48:09.820 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:48:09.820 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:48:09.820 139328 ERROR oslo_service.periodic_task >2018-06-25 02:48:39.691 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:48:39.692 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:48:39.698 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:48:39.699 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:48:39.733 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:48:39.733 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:48:39.733 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:48:39.733 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:48:39.733 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:48:39.733 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:48:39.733 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:48:39.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:48:39.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:48:39.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:48:49.736 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:48:49.771 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:48:49.771 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:48:49.771 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:48:49.771 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:48:49.771 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:48:49.771 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:48:49.771 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:48:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:48:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:48:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:49:09.774 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:49:09.806 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:49:09.806 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:49:09.806 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:49:09.806 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:49:09.806 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:49:09.806 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:49:09.806 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:49:09.808 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:49:09.809 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:49:09.809 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:49:09.809 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:49:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:49:09.809 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:49:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:49:09.809 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:49:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:49:09.809 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:49:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:49:09.809 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:49:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:49:09.809 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:49:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:49:09.809 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:49:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:49:09.809 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:49:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:49:09.809 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:49:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:49:09.809 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:49:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:49:09.809 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:49:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:49:09.809 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:49:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:49:09.809 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:49:09.809 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:49:09.809 139328 ERROR oslo_service.periodic_task >2018-06-25 02:49:39.690 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:49:39.691 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:49:39.699 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:49:39.700 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:49:39.735 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:49:39.735 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:49:39.735 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:49:39.735 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:49:39.735 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:49:39.735 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:49:39.735 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:49:39.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:49:39.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:49:39.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:49:49.739 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:49:49.773 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:49:49.773 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:49:49.773 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:49:49.773 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:49:49.773 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:49:49.773 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:49:49.773 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:49:49.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:49:49.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:49:49.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:50:09.776 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:50:09.811 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:50:09.811 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:50:09.811 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:50:09.811 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:50:09.811 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:50:09.811 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:50:09.811 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:50:09.812 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:50:09.813 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:50:09.813 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:50:09.813 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:50:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:50:09.813 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:50:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:50:09.813 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:50:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:50:09.813 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:50:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:50:09.813 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:50:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:50:09.813 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:50:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:50:09.813 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:50:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:50:09.813 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:50:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:50:09.813 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:50:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:50:09.813 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:50:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:50:09.813 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:50:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:50:09.813 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:50:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:50:09.813 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:50:09.813 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:50:09.813 139328 ERROR oslo_service.periodic_task >2018-06-25 02:50:39.695 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:50:39.695 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:50:39.702 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:50:39.702 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:50:39.733 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:50:39.733 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:50:39.733 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:50:39.733 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:50:39.733 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:50:39.733 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:50:39.733 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:50:39.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:50:39.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.032 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:50:39.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:50:49.736 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:50:49.771 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:50:49.771 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:50:49.771 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:50:49.771 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:50:49.771 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:50:49.771 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:50:49.771 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:50:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:50:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.07 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:50:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:51:09.774 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:51:09.807 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:51:09.807 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:51:09.807 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:51:09.807 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:51:09.807 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:51:09.807 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:51:09.807 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:51:09.809 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:51:09.809 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.106 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:51:09.809 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:51:09.809 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:51:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:51:09.809 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:51:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:51:09.809 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:51:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:51:09.809 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:51:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:51:09.809 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:51:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:51:09.809 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:51:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:51:09.809 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:51:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:51:09.809 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:51:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:51:09.809 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:51:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:51:09.809 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:51:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:51:09.809 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:51:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:51:09.809 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:51:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:51:09.809 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:51:09.809 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:51:09.809 139328 ERROR oslo_service.periodic_task >2018-06-25 02:51:39.691 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:51:39.692 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:51:39.698 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:51:39.698 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:51:39.733 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:51:39.733 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:51:39.733 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:51:39.733 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:51:39.733 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:51:39.733 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:51:39.733 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:51:39.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:51:39.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:51:39.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:51:49.736 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:51:49.770 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:51:49.770 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:51:49.770 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:51:49.770 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:51:49.770 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:51:49.770 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:51:49.770 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:51:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:51:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:51:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:52:09.774 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:52:09.807 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:52:09.807 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:52:09.807 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:52:09.807 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:52:09.807 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:52:09.807 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:52:09.807 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:52:09.809 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:52:09.809 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:52:09.809 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:52:09.809 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:52:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:52:09.809 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:52:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:52:09.809 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:52:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:52:09.809 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:52:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:52:09.809 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:52:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:52:09.809 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:52:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:52:09.809 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:52:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:52:09.809 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:52:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:52:09.809 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:52:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:52:09.809 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:52:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:52:09.809 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:52:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:52:09.809 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:52:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:52:09.809 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:52:09.809 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:52:09.809 139328 ERROR oslo_service.periodic_task >2018-06-25 02:52:39.691 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:52:39.692 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:52:39.698 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:52:39.699 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:52:39.734 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:52:39.734 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:52:39.734 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:52:39.734 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:52:39.734 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:52:39.734 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:52:39.734 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:52:39.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:52:39.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:52:39.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:52:49.737 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:52:49.774 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:52:49.774 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:52:49.774 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:52:49.774 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:52:49.774 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:52:49.774 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:52:49.774 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:52:49.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:52:49.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:52:49.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:53:09.777 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:53:09.812 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:53:09.812 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:53:09.812 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:53:09.812 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:53:09.812 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:53:09.812 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:53:09.812 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:53:09.814 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:53:09.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:53:09.815 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:53:09.815 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:53:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:53:09.815 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:53:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:53:09.815 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:53:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:53:09.815 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:53:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:53:09.815 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:53:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:53:09.815 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:53:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:53:09.815 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:53:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:53:09.815 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:53:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:53:09.815 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:53:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:53:09.815 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:53:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:53:09.815 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:53:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:53:09.815 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:53:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:53:09.815 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:53:09.815 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:53:09.815 139328 ERROR oslo_service.periodic_task >2018-06-25 02:53:39.686 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:53:39.687 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:53:39.694 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:53:39.695 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:53:39.733 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:53:39.733 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:53:39.733 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:53:39.733 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:53:39.733 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:53:39.733 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:53:39.733 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:53:39.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:53:39.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.041 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:53:39.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:53:49.737 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:53:49.772 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:53:49.772 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:53:49.772 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:53:49.772 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:53:49.772 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:53:49.772 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:53:49.772 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:53:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:53:49.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.08 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:53:49.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:54:09.775 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:54:09.822 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:54:09.822 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:54:09.822 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:54:09.822 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:54:09.822 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:54:09.822 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:54:09.822 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:54:09.824 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:54:09.824 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.129 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:54:09.825 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:54:09.825 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:54:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:54:09.825 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:54:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:54:09.825 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:54:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:54:09.825 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:54:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:54:09.825 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:54:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:54:09.825 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:54:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:54:09.825 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:54:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:54:09.825 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:54:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:54:09.825 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:54:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:54:09.825 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:54:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:54:09.825 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:54:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:54:09.825 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:54:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:54:09.825 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:54:09.825 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:54:09.825 139328 ERROR oslo_service.periodic_task >2018-06-25 02:54:39.686 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:54:39.687 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:54:39.694 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:54:39.694 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:54:39.729 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:54:39.729 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:54:39.729 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:54:39.729 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:54:39.729 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:54:39.729 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:54:39.729 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:54:39.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:54:39.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:54:39.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:54:49.732 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:54:49.768 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:54:49.768 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:54:49.768 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:54:49.768 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:54:49.768 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:54:49.768 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:54:49.768 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:54:49.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:54:49.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:54:49.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:55:09.771 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:55:09.809 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:55:09.809 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:55:09.809 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:55:09.809 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:55:09.809 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:55:09.809 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:55:09.809 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:55:09.811 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:55:09.811 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:55:09.811 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:55:09.811 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:55:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:55:09.811 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:55:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:55:09.811 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:55:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:55:09.811 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:55:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:55:09.811 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:55:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:55:09.811 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:55:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:55:09.811 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:55:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:55:09.811 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:55:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:55:09.811 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:55:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:55:09.811 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:55:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:55:09.811 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:55:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:55:09.811 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:55:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:55:09.811 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:55:09.811 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:55:09.811 139328 ERROR oslo_service.periodic_task >2018-06-25 02:55:39.683 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:55:39.684 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:55:39.717 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:55:39.717 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:55:39.717 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:55:39.717 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:55:39.717 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:55:39.717 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:55:39.717 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:55:39.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:55:39.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:55:39.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:55:49.721 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:55:49.757 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:55:49.757 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:55:49.757 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:55:49.757 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:55:49.757 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:55:49.757 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:55:49.757 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:55:49.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:55:49.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:55:49.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:56:09.760 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:56:09.793 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:56:09.793 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:56:09.793 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:56:09.793 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:56:09.793 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:56:09.793 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:56:09.793 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:56:09.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:56:09.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:56:09.797 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:56:09.797 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:56:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:56:09.797 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:56:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:56:09.797 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:56:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:56:09.797 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:56:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:56:09.797 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:56:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:56:09.797 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:56:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:56:09.797 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:56:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:56:09.797 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:56:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:56:09.797 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:56:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:56:09.797 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:56:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:56:09.797 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:56:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:56:09.797 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:56:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:56:09.797 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:56:09.797 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:56:09.797 139328 ERROR oslo_service.periodic_task >2018-06-25 02:56:39.678 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:56:39.680 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:56:39.688 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:56:39.689 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:56:39.745 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:56:39.745 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:56:39.745 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:56:39.745 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:56:39.745 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:56:39.745 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:56:39.745 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:56:39.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:56:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.061 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:56:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:56:49.750 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:56:49.784 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:56:49.784 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:56:49.784 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:56:49.784 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:56:49.784 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:56:49.784 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:56:49.784 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:56:49.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:56:49.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.098 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:56:49.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:57:09.787 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:57:09.820 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:57:09.820 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:57:09.820 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:57:09.820 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:57:09.820 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:57:09.820 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:57:09.820 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:57:09.822 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:57:09.822 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.133 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:57:09.822 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:57:09.822 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:57:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:57:09.822 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:57:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:57:09.822 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:57:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:57:09.822 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:57:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:57:09.822 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:57:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:57:09.822 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:57:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:57:09.822 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:57:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:57:09.822 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:57:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:57:09.822 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:57:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:57:09.822 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:57:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:57:09.822 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:57:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:57:09.822 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:57:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:57:09.822 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:57:09.822 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:57:09.822 139328 ERROR oslo_service.periodic_task >2018-06-25 02:57:39.674 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:57:39.674 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:57:39.682 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:57:39.683 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:57:39.718 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:57:39.718 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:57:39.718 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:57:39.718 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:57:39.718 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:57:39.718 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:57:39.718 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:57:39.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:57:39.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:57:39.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:57:49.722 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:57:49.759 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:57:49.759 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:57:49.759 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:57:49.759 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:57:49.759 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:57:49.759 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:57:49.759 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:57:49.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:57:49.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:57:49.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:58:09.762 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:58:09.799 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:58:09.799 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:58:09.799 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:58:09.799 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:58:09.799 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:58:09.799 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:58:09.799 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:58:09.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:58:09.801 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.118 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:58:09.801 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:58:09.801 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:58:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:58:09.801 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:58:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:58:09.801 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:58:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:58:09.801 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:58:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:58:09.801 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:58:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:58:09.801 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:58:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:58:09.801 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:58:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:58:09.801 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:58:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:58:09.801 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:58:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:58:09.801 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:58:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:58:09.801 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:58:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:58:09.801 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:58:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:58:09.801 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:58:09.801 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:58:09.801 139328 ERROR oslo_service.periodic_task >2018-06-25 02:58:39.672 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:58:39.673 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:58:39.682 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:58:39.683 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:58:39.716 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:58:39.716 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:58:39.716 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:58:39.716 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:58:39.716 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:58:39.716 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:58:39.716 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:58:39.718 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:58:39.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:58:39.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:58:49.719 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:58:49.754 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:58:49.754 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:58:49.754 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:58:49.754 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:58:49.754 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:58:49.754 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:58:49.754 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:58:49.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:58:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:58:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:59:09.757 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:59:09.791 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:59:09.791 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:59:09.791 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:59:09.791 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:59:09.791 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:59:09.791 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:59:09.791 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:59:09.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:59:09.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:59:09.795 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:59:09.795 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 02:59:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 02:59:09.795 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 02:59:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 02:59:09.795 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 02:59:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 02:59:09.795 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 02:59:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 02:59:09.795 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 02:59:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 02:59:09.795 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 02:59:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 02:59:09.795 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 02:59:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 02:59:09.795 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 02:59:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 02:59:09.795 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 02:59:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 02:59:09.795 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 02:59:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 02:59:09.795 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 02:59:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 02:59:09.795 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 02:59:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 02:59:09.795 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 02:59:09.795 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 02:59:09.795 139328 ERROR oslo_service.periodic_task >2018-06-25 02:59:39.677 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:59:39.678 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 02:59:39.684 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 02:59:39.685 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:59:39.719 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:59:39.719 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:59:39.719 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:59:39.719 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:59:39.719 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:59:39.719 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:59:39.719 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:59:39.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:59:39.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:59:39.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 02:59:49.723 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 02:59:49.761 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 02:59:49.761 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 02:59:49.761 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 02:59:49.761 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 02:59:49.761 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 02:59:49.761 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 02:59:49.761 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 02:59:49.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 02:59:49.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 02:59:49.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:00:09.764 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:00:09.803 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:00:09.803 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:00:09.803 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:00:09.803 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:00:09.803 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:00:09.803 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:00:09.803 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:00:09.805 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:00:09.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.121 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:00:09.806 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:00:09.806 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:00:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:00:09.806 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:00:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:00:09.806 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:00:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:00:09.806 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:00:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:00:09.806 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:00:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:00:09.806 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:00:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:00:09.806 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:00:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:00:09.806 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:00:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:00:09.806 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:00:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:00:09.806 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:00:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:00:09.806 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:00:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:00:09.806 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:00:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:00:09.806 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:00:09.806 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:00:09.806 139328 ERROR oslo_service.periodic_task >2018-06-25 03:00:39.677 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:00:39.678 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:00:39.685 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:00:39.685 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:00:39.718 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:00:39.718 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:00:39.718 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:00:39.718 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:00:39.718 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:00:39.718 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:00:39.718 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:00:39.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:00:39.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:00:39.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:00:49.721 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:00:49.757 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:00:49.757 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:00:49.757 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:00:49.757 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:00:49.757 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:00:49.757 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:00:49.757 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:00:49.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:00:49.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:00:49.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:01:09.761 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:01:09.803 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:01:09.803 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:01:09.803 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:01:09.803 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:01:09.803 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:01:09.803 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:01:09.803 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:01:09.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:01:09.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.12 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:01:09.807 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:01:09.807 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:01:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:01:09.807 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:01:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:01:09.807 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:01:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:01:09.807 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:01:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:01:09.807 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:01:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:01:09.807 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:01:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:01:09.807 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:01:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:01:09.807 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:01:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:01:09.807 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:01:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:01:09.807 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:01:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:01:09.807 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:01:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:01:09.807 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:01:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:01:09.807 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:01:09.807 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:01:09.807 139328 ERROR oslo_service.periodic_task >2018-06-25 03:01:39.678 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:01:39.680 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:01:39.687 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:01:39.687 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:01:39.724 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:01:39.724 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:01:39.724 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:01:39.724 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:01:39.724 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:01:39.724 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:01:39.724 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:01:39.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:01:39.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:01:39.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:01:49.728 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:01:49.761 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:01:49.761 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:01:49.761 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:01:49.761 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:01:49.761 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:01:49.761 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:01:49.761 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:01:49.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:01:49.764 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:01:49.764 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:02:09.764 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:02:09.798 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:02:09.798 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:02:09.798 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:02:09.798 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:02:09.798 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:02:09.798 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:02:09.798 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:02:09.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:02:09.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:02:09.801 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:02:09.801 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:02:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:02:09.801 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:02:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:02:09.801 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:02:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:02:09.801 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:02:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:02:09.801 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:02:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:02:09.801 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:02:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:02:09.801 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:02:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:02:09.801 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:02:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:02:09.801 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:02:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:02:09.801 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:02:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:02:09.801 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:02:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:02:09.801 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:02:09.801 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:02:09.801 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:02:09.801 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:02:09.801 139328 ERROR oslo_service.periodic_task >2018-06-25 03:02:39.682 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:02:39.683 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:02:39.689 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:02:39.690 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:02:39.726 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:02:39.726 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:02:39.726 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:02:39.726 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:02:39.726 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:02:39.726 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:02:39.726 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:02:39.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:02:39.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:02:39.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:02:49.731 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:02:49.765 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:02:49.765 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:02:49.765 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:02:49.765 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:02:49.765 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:02:49.765 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:02:49.765 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:02:49.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:02:49.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:02:49.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:03:09.768 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:03:09.802 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:03:09.802 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:03:09.802 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:03:09.802 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:03:09.802 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:03:09.802 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:03:09.802 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:03:09.804 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:03:09.804 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:03:09.804 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:03:09.804 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:03:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:03:09.804 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:03:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:03:09.804 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:03:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:03:09.804 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:03:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:03:09.804 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:03:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:03:09.804 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:03:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:03:09.804 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:03:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:03:09.804 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:03:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:03:09.804 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:03:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:03:09.804 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:03:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:03:09.804 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:03:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:03:09.804 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:03:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:03:09.804 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:03:09.804 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:03:09.804 139328 ERROR oslo_service.periodic_task >2018-06-25 03:03:39.686 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:03:39.687 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:03:39.693 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:03:39.694 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:03:39.729 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:03:39.729 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:03:39.729 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:03:39.729 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:03:39.729 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:03:39.729 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:03:39.729 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:03:39.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:03:39.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:03:39.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:03:49.732 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:03:49.766 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:03:49.766 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:03:49.766 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:03:49.766 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:03:49.766 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:03:49.766 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:03:49.766 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:03:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:03:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:03:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:04:09.769 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:04:09.805 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:04:09.805 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:04:09.805 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:04:09.805 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:04:09.805 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:04:09.805 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:04:09.805 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:04:09.807 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:04:09.808 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:04:09.808 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:04:09.808 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:04:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:04:09.808 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:04:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:04:09.808 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:04:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:04:09.808 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:04:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:04:09.808 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:04:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:04:09.808 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:04:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:04:09.808 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:04:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:04:09.808 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:04:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:04:09.808 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:04:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:04:09.808 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:04:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:04:09.808 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:04:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:04:09.808 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:04:09.808 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:04:09.808 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:04:09.808 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:04:09.808 139328 ERROR oslo_service.periodic_task >2018-06-25 03:04:39.690 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:04:39.691 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:04:39.697 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:04:39.698 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:04:39.730 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:04:39.730 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:04:39.730 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:04:39.730 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:04:39.730 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:04:39.730 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:04:39.730 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:04:39.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:04:39.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:04:39.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:04:49.734 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:04:49.775 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:04:49.775 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:04:49.775 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:04:49.775 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:04:49.775 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:04:49.775 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:04:49.775 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:04:49.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:04:49.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:04:49.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:05:09.778 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:05:09.818 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:05:09.818 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:05:09.818 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:05:09.818 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:05:09.818 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:05:09.818 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:05:09.818 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:05:09.820 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:05:09.820 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.122 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:05:09.820 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:05:09.820 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:05:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:05:09.820 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:05:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:05:09.820 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:05:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:05:09.820 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:05:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:05:09.820 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:05:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:05:09.820 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:05:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:05:09.820 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:05:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:05:09.820 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:05:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:05:09.820 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:05:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:05:09.820 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:05:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:05:09.820 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:05:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:05:09.820 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:05:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:05:09.820 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:05:09.820 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:05:09.820 139328 ERROR oslo_service.periodic_task >2018-06-25 03:05:39.692 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:05:39.693 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:05:39.702 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:05:39.703 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:05:39.740 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:05:39.740 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:05:39.740 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:05:39.740 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:05:39.740 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:05:39.740 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:05:39.740 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:05:39.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:05:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:05:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:05:49.744 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:05:49.777 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:05:49.777 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:05:49.777 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:05:49.777 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:05:49.777 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:05:49.777 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:05:49.777 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:05:49.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:05:49.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:05:49.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:06:09.780 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:06:09.814 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:06:09.814 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:06:09.814 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:06:09.814 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:06:09.814 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:06:09.814 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:06:09.814 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:06:09.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:06:09.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:06:09.817 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:06:09.817 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:06:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:06:09.817 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:06:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:06:09.817 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:06:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:06:09.817 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:06:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:06:09.817 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:06:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:06:09.817 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:06:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:06:09.817 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:06:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:06:09.817 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:06:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:06:09.817 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:06:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:06:09.817 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:06:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:06:09.817 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:06:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:06:09.817 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:06:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:06:09.817 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:06:09.817 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:06:09.817 139328 ERROR oslo_service.periodic_task >2018-06-25 03:06:39.689 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:06:39.690 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:06:39.702 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:06:39.702 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:06:39.738 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:06:39.738 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:06:39.738 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:06:39.738 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:06:39.738 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:06:39.738 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:06:39.738 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:06:39.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:06:39.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:06:39.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:06:49.741 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:06:49.773 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:06:49.773 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:06:49.773 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:06:49.773 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:06:49.773 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:06:49.773 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:06:49.773 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:06:49.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:06:49.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:06:49.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:07:09.777 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:07:09.813 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:07:09.813 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:07:09.813 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:07:09.813 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:07:09.813 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:07:09.813 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:07:09.813 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:07:09.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:07:09.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:07:09.816 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:07:09.816 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:07:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:07:09.816 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:07:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:07:09.816 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:07:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:07:09.816 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:07:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:07:09.816 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:07:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:07:09.816 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:07:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:07:09.816 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:07:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:07:09.816 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:07:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:07:09.816 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:07:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:07:09.816 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:07:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:07:09.816 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:07:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:07:09.816 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:07:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:07:09.816 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:07:09.816 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:07:09.816 139328 ERROR oslo_service.periodic_task >2018-06-25 03:07:39.688 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:07:39.689 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:07:39.700 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:07:39.701 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:07:39.739 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:07:39.739 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:07:39.739 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:07:39.739 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:07:39.739 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:07:39.739 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:07:39.739 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:07:39.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:07:39.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.041 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:07:39.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:07:49.742 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:07:49.775 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:07:49.775 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:07:49.775 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:07:49.775 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:07:49.775 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:07:49.775 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:07:49.775 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:07:49.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:07:49.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:07:49.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:08:09.777 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:08:09.810 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:08:09.810 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:08:09.810 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:08:09.810 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:08:09.810 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:08:09.810 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:08:09.810 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:08:09.812 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:08:09.813 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:08:09.813 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:08:09.813 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:08:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:08:09.813 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:08:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:08:09.813 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:08:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:08:09.813 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:08:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:08:09.813 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:08:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:08:09.813 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:08:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:08:09.813 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:08:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:08:09.813 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:08:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:08:09.813 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:08:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:08:09.813 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:08:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:08:09.813 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:08:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:08:09.813 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:08:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:08:09.813 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:08:09.813 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:08:09.813 139328 ERROR oslo_service.periodic_task >2018-06-25 03:08:39.685 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:08:39.686 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:08:39.692 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:08:39.692 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:08:39.729 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:08:39.729 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:08:39.729 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:08:39.729 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:08:39.729 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:08:39.729 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:08:39.729 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:08:39.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:08:39.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:08:39.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:08:49.733 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:08:49.766 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:08:49.766 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:08:49.766 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:08:49.766 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:08:49.766 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:08:49.766 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:08:49.766 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:08:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:08:49.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:08:49.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:09:09.770 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:09:09.806 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:09:09.806 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:09:09.806 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:09:09.806 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:09:09.806 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:09:09.806 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:09:09.806 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:09:09.808 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:09:09.808 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:09:09.809 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:09:09.809 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:09:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:09:09.809 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:09:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:09:09.809 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:09:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:09:09.809 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:09:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:09:09.809 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:09:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:09:09.809 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:09:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:09:09.809 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:09:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:09:09.809 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:09:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:09:09.809 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:09:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:09:09.809 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:09:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:09:09.809 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:09:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:09:09.809 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:09:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:09:09.809 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:09:09.809 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:09:09.809 139328 ERROR oslo_service.periodic_task >2018-06-25 03:09:39.680 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:09:39.681 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:09:39.688 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:09:39.688 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:09:39.722 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:09:39.722 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:09:39.722 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:09:39.722 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:09:39.722 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:09:39.722 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:09:39.722 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:09:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:09:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:09:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:09:49.725 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:09:49.761 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:09:49.761 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:09:49.761 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:09:49.761 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:09:49.761 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:09:49.761 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:09:49.761 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:09:49.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:09:49.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:09:49.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:10:09.764 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:10:09.804 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:10:09.804 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:10:09.804 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:10:09.804 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:10:09.804 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:10:09.804 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:10:09.804 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:10:09.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:10:09.807 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.118 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:10:09.807 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:10:09.807 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:10:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:10:09.807 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:10:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:10:09.807 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:10:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:10:09.807 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:10:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:10:09.807 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:10:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:10:09.807 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:10:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:10:09.807 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:10:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:10:09.807 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:10:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:10:09.807 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:10:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:10:09.807 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:10:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:10:09.807 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:10:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:10:09.807 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:10:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:10:09.807 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:10:09.807 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:10:09.807 139328 ERROR oslo_service.periodic_task >2018-06-25 03:10:39.678 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:10:39.679 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:10:39.685 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:10:39.686 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:10:39.719 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:10:39.719 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:10:39.719 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:10:39.719 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:10:39.719 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:10:39.719 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:10:39.719 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:10:39.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:10:39.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:10:39.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:10:49.722 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:10:49.756 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:10:49.756 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:10:49.756 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:10:49.756 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:10:49.756 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:10:49.756 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:10:49.756 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:10:49.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:10:49.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:10:49.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:11:09.759 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:11:09.794 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:11:09.794 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:11:09.794 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:11:09.794 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:11:09.794 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:11:09.794 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:11:09.794 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:11:09.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:11:09.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:11:09.797 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:11:09.797 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:11:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:11:09.797 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:11:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:11:09.797 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:11:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:11:09.797 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:11:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:11:09.797 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:11:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:11:09.797 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:11:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:11:09.797 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:11:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:11:09.797 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:11:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:11:09.797 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:11:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:11:09.797 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:11:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:11:09.797 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:11:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:11:09.797 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:11:09.797 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:11:09.797 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:11:09.797 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:11:09.797 139328 ERROR oslo_service.periodic_task >2018-06-25 03:11:39.678 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:11:39.679 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:11:39.686 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:11:39.687 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:11:39.727 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:11:39.727 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:11:39.727 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:11:39.727 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:11:39.727 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:11:39.727 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:11:39.727 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:11:39.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:11:39.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.042 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:11:39.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:11:49.731 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:11:49.766 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:11:49.766 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:11:49.766 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:11:49.766 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:11:49.766 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:11:49.766 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:11:49.766 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:11:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:11:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.081 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:11:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:12:09.770 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:12:09.808 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:12:09.808 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:12:09.808 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:12:09.808 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:12:09.808 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:12:09.808 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:12:09.808 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:12:09.810 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:12:09.810 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.123 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:12:09.811 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:12:09.811 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:12:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:12:09.811 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:12:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:12:09.811 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:12:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:12:09.811 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:12:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:12:09.811 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:12:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:12:09.811 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:12:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:12:09.811 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:12:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:12:09.811 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:12:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:12:09.811 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:12:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:12:09.811 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:12:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:12:09.811 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:12:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:12:09.811 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:12:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:12:09.811 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:12:09.811 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:12:09.811 139328 ERROR oslo_service.periodic_task >2018-06-25 03:12:39.682 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:12:39.683 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:12:39.689 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:12:39.690 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:12:39.736 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:12:39.736 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:12:39.736 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:12:39.736 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:12:39.736 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:12:39.736 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:12:39.736 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:12:39.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:12:39.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.048 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:12:39.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:12:49.739 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:12:49.773 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:12:49.773 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:12:49.773 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:12:49.773 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:12:49.773 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:12:49.773 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:12:49.773 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:12:49.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:12:49.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.084 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:12:49.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:13:09.776 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:13:09.808 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:13:09.808 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:13:09.808 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:13:09.808 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:13:09.808 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:13:09.808 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:13:09.808 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:13:09.810 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:13:09.810 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.119 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:13:09.811 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:13:09.811 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:13:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:13:09.811 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:13:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:13:09.811 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:13:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:13:09.811 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:13:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:13:09.811 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:13:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:13:09.811 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:13:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:13:09.811 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:13:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:13:09.811 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:13:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:13:09.811 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:13:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:13:09.811 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:13:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:13:09.811 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:13:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:13:09.811 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:13:09.811 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:13:09.811 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:13:09.811 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:13:09.811 139328 ERROR oslo_service.periodic_task >2018-06-25 03:13:39.682 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:13:39.683 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:13:39.690 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:13:39.690 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:13:39.723 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:13:39.723 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:13:39.723 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:13:39.723 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:13:39.723 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:13:39.723 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:13:39.723 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:13:39.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:13:39.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:13:39.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:13:49.726 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:13:49.761 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:13:49.761 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:13:49.761 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:13:49.761 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:13:49.761 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:13:49.761 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:13:49.761 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:13:49.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:13:49.764 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:13:49.764 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:14:09.764 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:14:09.797 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:14:09.797 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:14:09.797 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:14:09.797 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:14:09.797 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:14:09.797 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:14:09.797 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:14:09.799 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:14:09.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:14:09.800 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:14:09.800 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:14:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:14:09.800 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:14:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:14:09.800 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:14:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:14:09.800 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:14:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:14:09.800 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:14:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:14:09.800 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:14:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:14:09.800 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:14:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:14:09.800 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:14:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:14:09.800 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:14:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:14:09.800 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:14:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:14:09.800 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:14:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:14:09.800 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:14:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:14:09.800 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:14:09.800 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:14:09.800 139328 ERROR oslo_service.periodic_task >2018-06-25 03:14:39.682 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:14:39.683 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:14:39.689 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:14:39.690 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:14:39.725 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:14:39.725 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:14:39.725 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:14:39.725 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:14:39.725 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:14:39.725 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:14:39.725 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:14:39.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:14:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:14:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:14:49.729 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:14:49.769 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:14:49.769 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:14:49.769 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:14:49.769 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:14:49.769 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:14:49.769 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:14:49.769 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:14:49.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:14:49.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.081 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:14:49.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:15:09.772 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:15:09.807 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:15:09.807 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:15:09.807 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:15:09.807 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:15:09.807 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:15:09.807 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:15:09.807 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:15:09.809 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:15:09.809 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.12 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:15:09.810 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:15:09.810 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:15:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:15:09.810 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:15:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:15:09.810 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:15:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:15:09.810 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:15:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:15:09.810 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:15:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:15:09.810 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:15:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:15:09.810 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:15:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:15:09.810 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:15:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:15:09.810 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:15:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:15:09.810 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:15:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:15:09.810 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:15:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:15:09.810 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:15:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:15:09.810 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:15:09.810 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:15:09.810 139328 ERROR oslo_service.periodic_task >2018-06-25 03:15:39.681 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:15:39.682 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:15:39.691 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:15:39.692 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:15:39.726 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:15:39.726 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:15:39.726 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:15:39.726 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:15:39.726 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:15:39.726 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:15:39.726 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:15:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:15:39.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:15:39.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:15:49.729 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:15:49.764 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:15:49.764 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:15:49.764 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:15:49.764 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:15:49.764 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:15:49.764 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:15:49.764 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:15:49.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:15:49.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:15:49.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:16:09.767 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:16:09.803 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:16:09.803 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:16:09.803 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:16:09.803 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:16:09.803 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:16:09.803 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:16:09.803 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:16:09.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:16:09.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:16:09.807 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:16:09.807 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:16:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:16:09.807 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:16:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:16:09.807 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:16:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:16:09.807 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:16:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:16:09.807 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:16:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:16:09.807 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:16:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:16:09.807 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:16:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:16:09.807 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:16:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:16:09.807 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:16:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:16:09.807 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:16:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:16:09.807 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:16:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:16:09.807 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:16:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:16:09.807 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:16:09.807 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:16:09.807 139328 ERROR oslo_service.periodic_task >2018-06-25 03:16:39.678 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:16:39.679 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:16:39.687 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:16:39.688 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:16:39.721 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:16:39.721 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:16:39.721 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:16:39.721 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:16:39.721 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:16:39.721 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:16:39.721 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:16:39.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:16:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:16:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:16:49.725 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:16:49.783 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:16:49.783 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:16:49.783 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:16:49.783 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:16:49.783 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:16:49.783 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:16:49.783 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:16:49.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:16:49.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.098 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:16:49.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:17:09.786 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:17:09.840 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:17:09.840 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:17:09.840 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:17:09.840 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:17:09.840 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:17:09.840 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:17:09.840 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:17:09.842 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:17:09.842 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.155 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:17:09.843 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:17:09.843 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:17:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:17:09.843 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:17:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:17:09.843 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:17:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:17:09.843 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:17:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:17:09.843 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:17:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:17:09.843 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:17:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:17:09.843 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:17:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:17:09.843 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:17:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:17:09.843 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:17:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:17:09.843 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:17:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:17:09.843 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:17:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:17:09.843 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:17:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:17:09.843 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:17:09.843 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:17:09.843 139328 ERROR oslo_service.periodic_task >2018-06-25 03:17:39.674 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:17:39.676 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:17:39.683 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:17:39.684 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:17:39.728 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:17:39.728 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:17:39.728 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:17:39.728 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:17:39.728 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:17:39.728 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:17:39.728 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:17:39.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:17:39.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.047 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:17:39.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:17:49.732 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:17:49.772 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:17:49.772 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:17:49.772 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:17:49.772 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:17:49.772 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:17:49.772 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:17:49.772 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:17:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:17:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.09 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:17:49.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:18:09.775 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:18:09.809 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:18:09.809 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:18:09.809 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:18:09.809 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:18:09.809 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:18:09.809 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:18:09.809 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:18:09.811 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:18:09.812 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.128 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:18:09.812 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:18:09.812 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:18:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:18:09.812 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:18:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:18:09.812 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:18:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:18:09.812 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:18:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:18:09.812 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:18:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:18:09.812 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:18:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:18:09.812 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:18:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:18:09.812 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:18:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:18:09.812 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:18:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:18:09.812 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:18:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:18:09.812 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:18:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:18:09.812 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:18:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:18:09.812 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:18:09.812 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:18:09.812 139328 ERROR oslo_service.periodic_task >2018-06-25 03:18:39.674 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:18:39.674 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:18:39.681 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:18:39.682 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:18:39.719 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:18:39.719 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:18:39.719 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:18:39.719 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:18:39.719 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:18:39.719 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:18:39.719 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:18:39.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:18:39.721 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:18:39.722 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:18:49.722 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:18:49.763 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:18:49.763 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:18:49.763 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:18:49.763 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:18:49.763 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:18:49.763 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:18:49.763 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:18:49.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:18:49.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.084 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:18:49.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:19:09.767 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:19:09.803 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:19:09.803 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:19:09.803 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:19:09.803 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:19:09.803 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:19:09.803 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:19:09.803 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:19:09.805 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:19:09.805 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.123 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:19:09.806 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:19:09.806 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:19:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:19:09.806 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:19:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:19:09.806 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:19:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:19:09.806 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:19:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:19:09.806 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:19:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:19:09.806 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:19:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:19:09.806 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:19:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:19:09.806 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:19:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:19:09.806 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:19:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:19:09.806 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:19:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:19:09.806 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:19:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:19:09.806 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:19:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:19:09.806 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:19:09.806 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:19:09.806 139328 ERROR oslo_service.periodic_task >2018-06-25 03:19:39.677 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:19:39.678 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:19:39.685 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:19:39.685 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:19:39.721 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:19:39.721 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:19:39.721 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:19:39.721 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:19:39.721 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:19:39.721 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:19:39.721 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:19:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:19:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:19:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:19:49.725 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:19:49.760 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:19:49.760 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:19:49.760 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:19:49.760 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:19:49.760 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:19:49.760 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:19:49.760 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:19:49.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:19:49.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:19:49.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:20:09.764 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:20:09.799 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:20:09.799 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:20:09.799 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:20:09.799 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:20:09.799 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:20:09.799 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:20:09.799 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:20:09.801 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:20:09.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:20:09.802 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:20:09.802 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:20:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:20:09.802 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:20:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:20:09.802 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:20:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:20:09.802 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:20:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:20:09.802 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:20:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:20:09.802 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:20:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:20:09.802 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:20:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:20:09.802 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:20:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:20:09.802 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:20:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:20:09.802 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:20:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:20:09.802 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:20:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:20:09.802 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:20:09.802 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:20:09.802 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:20:09.802 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:20:09.802 139328 ERROR oslo_service.periodic_task >2018-06-25 03:20:39.673 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:20:39.674 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:20:39.681 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:20:39.681 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:20:39.714 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:20:39.714 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:20:39.714 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:20:39.714 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:20:39.714 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:20:39.714 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:20:39.714 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:20:39.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:20:39.716 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:20:39.717 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:20:49.718 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:20:49.753 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:20:49.753 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:20:49.753 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:20:49.753 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:20:49.753 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:20:49.753 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:20:49.753 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:20:49.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:20:49.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:20:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:21:09.758 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:21:09.792 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:21:09.792 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:21:09.792 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:21:09.792 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:21:09.792 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:21:09.792 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:21:09.792 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:21:09.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:21:09.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:21:09.795 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:21:09.795 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:21:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:21:09.795 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:21:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:21:09.795 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:21:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:21:09.795 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:21:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:21:09.795 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:21:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:21:09.795 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:21:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:21:09.795 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:21:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:21:09.795 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:21:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:21:09.795 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:21:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:21:09.795 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:21:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:21:09.795 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:21:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:21:09.795 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:21:09.795 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:21:09.795 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:21:09.795 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:21:09.795 139328 ERROR oslo_service.periodic_task >2018-06-25 03:21:39.676 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:21:39.677 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:21:39.683 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:21:39.684 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:21:39.718 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:21:39.718 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:21:39.718 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:21:39.718 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:21:39.718 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:21:39.718 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:21:39.718 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:21:39.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:21:39.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:21:39.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:21:49.720 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:21:49.767 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:21:49.767 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:21:49.767 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:21:49.767 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:21:49.767 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:21:49.767 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:21:49.767 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:21:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:21:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.085 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:21:49.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:22:09.770 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:22:09.805 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:22:09.805 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:22:09.805 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:22:09.805 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:22:09.805 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:22:09.805 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:22:09.805 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:22:09.808 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:22:09.808 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.124 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:22:09.809 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:22:09.809 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:22:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:22:09.809 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:22:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:22:09.809 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:22:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:22:09.809 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:22:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:22:09.809 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:22:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:22:09.809 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:22:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:22:09.809 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:22:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:22:09.809 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:22:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:22:09.809 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:22:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:22:09.809 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:22:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:22:09.809 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:22:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:22:09.809 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:22:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:22:09.809 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:22:09.809 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:22:09.809 139328 ERROR oslo_service.periodic_task >2018-06-25 03:22:39.681 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:22:39.681 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:22:39.689 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:22:39.689 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:22:39.724 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:22:39.724 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:22:39.724 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:22:39.724 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:22:39.724 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:22:39.724 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:22:39.724 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:22:39.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:22:39.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:22:39.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:22:49.728 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:22:49.766 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:22:49.766 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:22:49.766 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:22:49.766 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:22:49.766 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:22:49.766 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:22:49.766 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:22:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:22:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:22:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:23:09.770 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:23:09.803 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:23:09.803 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:23:09.803 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:23:09.803 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:23:09.803 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:23:09.803 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:23:09.803 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:23:09.805 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:23:09.805 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:23:09.805 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:23:09.805 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:23:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:23:09.805 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:23:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:23:09.805 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:23:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:23:09.805 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:23:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:23:09.805 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:23:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:23:09.805 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:23:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:23:09.805 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:23:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:23:09.805 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:23:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:23:09.805 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:23:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:23:09.805 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:23:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:23:09.805 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:23:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:23:09.805 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:23:09.805 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:23:09.805 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:23:09.805 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:23:09.805 139328 ERROR oslo_service.periodic_task >2018-06-25 03:23:39.677 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:23:39.677 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:23:39.684 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:23:39.684 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:23:39.716 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:23:39.716 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:23:39.716 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:23:39.716 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:23:39.716 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:23:39.716 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:23:39.716 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:23:39.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:23:39.719 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:23:39.720 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:23:49.720 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:23:49.755 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:23:49.755 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:23:49.755 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:23:49.755 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:23:49.755 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:23:49.755 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:23:49.755 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:23:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:23:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:23:49.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:24:09.758 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:24:09.792 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:24:09.792 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:24:09.792 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:24:09.792 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:24:09.792 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:24:09.792 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:24:09.792 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:24:09.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:24:09.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:24:09.794 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:24:09.794 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:24:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:24:09.794 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:24:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:24:09.794 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:24:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:24:09.794 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:24:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:24:09.794 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:24:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:24:09.794 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:24:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:24:09.794 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:24:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:24:09.794 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:24:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:24:09.794 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:24:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:24:09.794 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:24:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:24:09.794 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:24:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:24:09.794 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:24:09.794 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:24:09.794 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:24:09.794 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:24:09.794 139328 ERROR oslo_service.periodic_task >2018-06-25 03:24:39.675 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:24:39.676 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:24:39.686 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:24:39.687 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:24:39.723 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:24:39.723 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:24:39.723 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:24:39.723 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:24:39.723 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:24:39.723 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:24:39.723 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:24:39.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:24:39.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:24:39.727 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:24:49.728 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:24:49.762 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:24:49.762 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:24:49.762 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:24:49.762 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:24:49.762 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:24:49.762 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:24:49.762 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:24:49.764 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:24:49.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:24:49.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:25:09.766 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:25:09.802 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:25:09.802 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:25:09.802 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:25:09.802 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:25:09.802 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:25:09.802 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:25:09.802 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:25:09.804 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:25:09.804 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:25:09.804 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:25:09.804 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:25:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:25:09.804 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:25:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:25:09.804 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:25:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:25:09.804 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:25:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:25:09.804 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:25:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:25:09.804 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:25:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:25:09.804 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:25:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:25:09.804 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:25:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:25:09.804 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:25:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:25:09.804 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:25:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:25:09.804 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:25:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:25:09.804 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:25:09.804 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:25:09.804 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:25:09.804 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:25:09.804 139328 ERROR oslo_service.periodic_task >2018-06-25 03:25:39.675 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:25:39.676 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:25:39.684 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:25:39.685 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:25:39.723 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:25:39.723 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:25:39.723 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:25:39.723 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:25:39.723 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:25:39.723 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:25:39.723 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:25:39.725 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:25:39.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.041 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:25:39.726 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:25:49.727 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:25:49.759 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:25:49.759 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:25:49.759 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:25:49.759 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:25:49.759 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:25:49.759 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:25:49.759 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:25:49.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:25:49.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:25:49.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:26:09.762 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:26:09.795 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:26:09.795 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:26:09.795 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:26:09.795 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:26:09.795 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:26:09.795 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:26:09.795 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:26:09.798 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:26:09.799 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:26:09.799 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:26:09.799 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:26:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:26:09.799 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:26:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:26:09.799 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:26:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:26:09.799 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:26:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:26:09.799 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:26:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:26:09.799 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:26:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:26:09.799 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:26:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:26:09.799 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:26:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:26:09.799 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:26:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:26:09.799 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:26:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:26:09.799 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:26:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:26:09.799 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:26:09.799 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:26:09.799 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:26:09.799 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:26:09.799 139328 ERROR oslo_service.periodic_task >2018-06-25 03:26:39.680 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:26:39.682 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:26:39.695 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:26:39.696 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:26:39.733 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:26:39.733 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:26:39.733 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:26:39.733 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:26:39.733 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:26:39.733 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:26:39.733 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:26:39.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:26:39.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:26:39.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:26:49.736 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:26:49.771 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:26:49.771 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:26:49.771 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:26:49.771 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:26:49.771 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:26:49.771 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:26:49.771 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:26:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:26:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:26:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:27:09.775 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:27:09.810 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:27:09.810 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:27:09.810 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:27:09.810 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:27:09.810 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:27:09.810 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:27:09.810 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:27:09.812 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:27:09.813 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:27:09.813 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:27:09.813 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:27:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:27:09.813 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:27:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:27:09.813 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:27:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:27:09.813 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:27:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:27:09.813 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:27:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:27:09.813 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:27:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:27:09.813 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:27:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:27:09.813 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:27:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:27:09.813 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:27:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:27:09.813 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:27:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:27:09.813 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:27:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:27:09.813 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:27:09.813 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:27:09.813 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:27:09.813 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:27:09.813 139328 ERROR oslo_service.periodic_task >2018-06-25 03:27:39.684 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:27:39.685 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:27:39.692 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:27:39.692 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:27:39.726 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:27:39.726 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:27:39.726 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:27:39.726 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:27:39.726 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:27:39.726 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:27:39.726 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:27:39.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:27:39.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:27:39.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:27:49.730 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:27:49.769 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:27:49.769 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:27:49.769 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:27:49.769 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:27:49.769 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:27:49.769 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:27:49.769 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:27:49.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:27:49.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:27:49.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:28:09.773 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:28:09.805 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:28:09.805 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:28:09.805 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:28:09.805 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:28:09.805 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:28:09.805 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:28:09.805 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:28:09.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:28:09.807 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:28:09.807 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:28:09.807 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:28:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:28:09.807 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:28:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:28:09.807 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:28:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:28:09.807 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:28:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:28:09.807 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:28:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:28:09.807 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:28:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:28:09.807 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:28:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:28:09.807 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:28:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:28:09.807 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:28:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:28:09.807 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:28:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:28:09.807 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:28:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:28:09.807 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:28:09.807 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:28:09.807 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:28:09.807 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:28:09.807 139328 ERROR oslo_service.periodic_task >2018-06-25 03:28:39.689 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:28:39.689 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:28:39.697 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:28:39.698 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:28:39.731 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:28:39.731 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:28:39.731 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:28:39.731 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:28:39.731 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:28:39.731 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:28:39.731 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:28:39.733 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:28:39.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:28:39.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:28:49.735 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:28:49.769 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:28:49.769 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:28:49.769 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:28:49.769 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:28:49.769 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:28:49.769 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:28:49.769 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:28:49.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:28:49.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:28:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:29:09.773 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:29:09.806 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:29:09.806 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:29:09.806 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:29:09.806 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:29:09.806 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:29:09.806 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:29:09.806 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:29:09.808 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:29:09.809 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:29:09.809 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:29:09.809 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:29:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:29:09.809 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:29:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:29:09.809 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:29:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:29:09.809 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:29:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:29:09.809 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:29:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:29:09.809 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:29:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:29:09.809 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:29:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:29:09.809 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:29:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:29:09.809 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:29:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:29:09.809 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:29:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:29:09.809 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:29:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:29:09.809 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:29:09.809 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:29:09.809 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:29:09.809 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:29:09.809 139328 ERROR oslo_service.periodic_task >2018-06-25 03:29:39.691 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:29:39.691 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:29:39.699 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:29:39.700 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:29:39.736 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:29:39.736 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:29:39.736 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:29:39.736 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:29:39.736 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:29:39.736 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:29:39.736 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:29:39.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:29:39.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:29:39.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:29:49.739 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:29:49.777 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:29:49.777 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:29:49.777 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:29:49.777 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:29:49.777 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:29:49.777 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:29:49.777 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:29:49.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:29:49.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:29:49.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:30:09.780 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:30:09.812 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:30:09.812 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:30:09.812 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:30:09.812 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:30:09.812 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:30:09.812 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:30:09.812 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:30:09.814 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:30:09.814 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:30:09.815 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:30:09.815 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:30:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:30:09.815 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:30:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:30:09.815 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:30:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:30:09.815 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:30:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:30:09.815 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:30:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:30:09.815 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:30:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:30:09.815 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:30:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:30:09.815 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:30:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:30:09.815 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:30:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:30:09.815 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:30:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:30:09.815 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:30:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:30:09.815 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:30:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:30:09.815 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:30:09.815 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:30:09.815 139328 ERROR oslo_service.periodic_task >2018-06-25 03:30:39.686 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:30:39.687 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:30:39.697 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:30:39.697 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:30:39.729 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:30:39.729 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:30:39.729 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:30:39.729 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:30:39.729 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:30:39.729 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:30:39.729 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:30:39.731 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:30:39.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:30:39.732 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:30:49.733 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:30:49.789 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:30:49.789 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:30:49.789 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:30:49.789 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:30:49.789 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:30:49.789 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:30:49.789 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:30:49.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:30:49.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.097 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:30:49.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:31:09.795 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:31:09.828 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:31:09.828 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:31:09.828 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:31:09.828 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:31:09.828 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:31:09.828 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:31:09.828 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:31:09.830 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:31:09.830 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.132 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:31:09.830 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:31:09.830 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:31:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:31:09.830 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:31:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:31:09.830 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:31:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:31:09.830 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:31:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:31:09.830 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:31:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:31:09.830 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:31:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:31:09.830 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:31:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:31:09.830 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:31:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:31:09.830 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:31:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:31:09.830 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:31:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:31:09.830 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:31:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:31:09.830 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:31:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:31:09.830 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:31:09.830 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:31:09.830 139328 ERROR oslo_service.periodic_task >2018-06-25 03:31:39.682 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:31:39.683 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:31:39.689 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:31:39.689 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:31:39.721 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:31:39.721 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:31:39.721 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:31:39.721 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:31:39.721 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:31:39.721 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:31:39.721 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:31:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:31:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:31:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:31:49.725 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:31:49.761 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:31:49.761 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:31:49.761 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:31:49.761 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:31:49.761 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:31:49.761 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:31:49.761 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:31:49.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:31:49.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:31:49.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:32:09.764 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:32:09.797 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:32:09.797 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:32:09.797 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:32:09.797 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:32:09.797 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:32:09.797 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:32:09.797 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:32:09.799 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:32:09.799 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:32:09.800 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:32:09.800 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:32:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:32:09.800 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:32:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:32:09.800 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:32:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:32:09.800 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:32:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:32:09.800 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:32:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:32:09.800 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:32:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:32:09.800 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:32:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:32:09.800 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:32:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:32:09.800 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:32:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:32:09.800 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:32:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:32:09.800 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:32:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:32:09.800 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:32:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:32:09.800 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:32:09.800 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:32:09.800 139328 ERROR oslo_service.periodic_task >2018-06-25 03:32:39.681 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:32:39.682 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:32:39.688 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:32:39.689 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:32:39.721 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:32:39.721 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:32:39.721 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:32:39.721 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:32:39.721 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:32:39.721 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:32:39.721 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:32:39.723 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:32:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:32:39.724 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:32:49.724 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:32:49.759 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:32:49.759 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:32:49.759 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:32:49.759 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:32:49.759 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:32:49.759 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:32:49.759 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:32:49.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:32:49.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:32:49.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:33:09.762 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:33:09.796 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:33:09.796 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:33:09.796 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:33:09.796 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:33:09.796 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:33:09.796 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:33:09.796 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:33:09.799 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:33:09.799 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:33:09.800 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:33:09.800 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:33:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:33:09.800 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:33:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:33:09.800 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:33:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:33:09.800 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:33:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:33:09.800 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:33:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:33:09.800 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:33:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:33:09.800 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:33:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:33:09.800 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:33:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:33:09.800 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:33:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:33:09.800 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:33:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:33:09.800 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:33:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:33:09.800 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:33:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:33:09.800 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:33:09.800 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:33:09.800 139328 ERROR oslo_service.periodic_task >2018-06-25 03:33:39.681 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:33:39.682 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:33:39.689 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:33:39.690 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:33:39.732 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:33:39.732 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:33:39.732 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:33:39.732 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:33:39.732 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:33:39.732 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:33:39.732 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:33:39.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:33:39.734 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.044 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:33:39.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:33:49.735 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:33:49.773 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:33:49.773 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:33:49.773 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:33:49.773 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:33:49.773 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:33:49.773 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:33:49.773 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:33:49.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:33:49.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.084 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:33:49.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:34:09.776 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:34:09.809 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:34:09.809 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:34:09.809 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:34:09.809 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:34:09.809 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:34:09.809 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:34:09.809 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:34:09.811 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:34:09.812 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.121 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:34:09.812 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:34:09.812 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:34:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:34:09.812 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:34:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:34:09.812 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:34:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:34:09.812 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:34:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:34:09.812 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:34:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:34:09.812 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:34:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:34:09.812 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:34:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:34:09.812 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:34:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:34:09.812 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:34:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:34:09.812 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:34:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:34:09.812 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:34:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:34:09.812 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:34:09.812 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:34:09.812 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:34:09.812 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:34:09.812 139328 ERROR oslo_service.periodic_task >2018-06-25 03:34:39.683 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:34:39.684 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:34:39.692 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:34:39.692 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:34:39.725 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:34:39.725 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:34:39.725 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:34:39.725 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:34:39.725 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:34:39.725 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:34:39.725 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:34:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:34:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:34:39.728 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:34:49.729 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:34:49.766 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:34:49.766 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:34:49.766 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:34:49.766 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:34:49.766 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:34:49.766 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:34:49.766 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:34:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:34:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:34:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:35:09.770 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:35:09.802 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:35:09.802 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:35:09.802 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:35:09.802 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:35:09.802 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:35:09.802 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:35:09.802 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:35:09.805 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:35:09.805 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:35:09.806 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:35:09.806 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:35:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:35:09.806 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:35:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:35:09.806 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:35:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:35:09.806 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:35:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:35:09.806 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:35:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:35:09.806 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:35:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:35:09.806 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:35:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:35:09.806 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:35:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:35:09.806 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:35:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:35:09.806 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:35:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:35:09.806 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:35:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:35:09.806 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:35:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:35:09.806 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:35:09.806 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:35:09.806 139328 ERROR oslo_service.periodic_task >2018-06-25 03:35:39.687 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:35:39.688 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:35:39.697 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:35:39.698 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:35:39.737 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:35:39.737 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:35:39.737 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:35:39.737 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:35:39.737 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:35:39.737 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:35:39.737 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:35:39.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:35:39.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.042 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:35:39.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:35:49.740 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:35:49.778 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:35:49.778 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:35:49.778 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:35:49.778 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:35:49.778 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:35:49.778 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:35:49.778 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:35:49.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:35:49.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.082 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:35:49.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:36:09.781 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:36:09.813 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:36:09.813 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:36:09.813 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:36:09.813 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:36:09.813 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:36:09.813 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:36:09.813 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:36:09.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:36:09.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:36:09.815 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:36:09.815 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:36:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:36:09.815 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:36:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:36:09.815 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:36:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:36:09.815 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:36:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:36:09.815 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:36:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:36:09.815 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:36:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:36:09.815 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:36:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:36:09.815 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:36:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:36:09.815 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:36:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:36:09.815 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:36:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:36:09.815 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:36:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:36:09.815 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:36:09.815 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:36:09.815 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:36:09.815 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:36:09.815 139328 ERROR oslo_service.periodic_task >2018-06-25 03:36:39.687 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:36:39.688 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:36:39.695 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:36:39.695 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:36:39.727 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:36:39.727 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:36:39.727 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:36:39.727 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:36:39.727 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:36:39.727 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:36:39.727 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:36:39.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:36:39.729 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:36:39.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:36:49.730 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:36:49.766 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:36:49.766 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:36:49.766 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:36:49.766 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:36:49.766 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:36:49.766 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:36:49.766 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:36:49.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:36:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:36:49.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:37:09.769 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:37:09.803 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:37:09.803 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:37:09.803 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:37:09.803 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:37:09.803 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:37:09.803 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:37:09.803 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:37:09.805 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:37:09.805 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:37:09.806 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:37:09.806 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:37:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:37:09.806 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:37:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:37:09.806 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:37:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:37:09.806 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:37:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:37:09.806 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:37:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:37:09.806 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:37:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:37:09.806 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:37:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:37:09.806 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:37:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:37:09.806 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:37:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:37:09.806 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:37:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:37:09.806 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:37:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:37:09.806 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:37:09.806 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:37:09.806 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:37:09.806 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:37:09.806 139328 ERROR oslo_service.periodic_task >2018-06-25 03:37:39.687 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:37:39.688 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:37:39.695 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:37:39.695 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:37:39.727 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:37:39.727 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:37:39.727 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:37:39.727 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:37:39.727 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:37:39.727 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:37:39.727 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:37:39.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:37:39.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:37:39.730 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:37:49.731 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:37:49.763 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:37:49.763 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:37:49.763 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:37:49.763 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:37:49.763 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:37:49.763 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:37:49.763 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:37:49.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:37:49.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.069 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:37:49.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:38:09.766 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:38:09.797 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:38:09.797 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:38:09.797 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:38:09.797 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:38:09.797 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:38:09.797 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:38:09.797 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:38:09.799 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:38:09.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.104 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:38:09.800 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:38:09.800 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:38:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:38:09.800 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:38:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:38:09.800 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:38:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:38:09.800 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:38:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:38:09.800 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:38:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:38:09.800 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:38:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:38:09.800 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:38:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:38:09.800 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:38:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:38:09.800 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:38:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:38:09.800 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:38:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:38:09.800 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:38:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:38:09.800 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:38:09.800 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:38:09.800 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:38:09.800 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:38:09.800 139328 ERROR oslo_service.periodic_task >2018-06-25 03:38:39.691 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:38:39.692 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:38:39.699 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:38:39.700 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:38:39.734 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:38:39.734 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:38:39.734 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:38:39.734 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:38:39.734 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:38:39.734 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:38:39.734 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:38:39.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:38:39.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:38:39.736 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:38:49.737 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:38:49.775 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:38:49.775 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:38:49.775 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:38:49.775 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:38:49.775 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:38:49.775 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:38:49.775 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:38:49.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:38:49.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:38:49.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:39:09.779 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:39:09.811 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:39:09.811 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:39:09.811 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:39:09.811 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:39:09.811 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:39:09.811 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:39:09.811 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:39:09.813 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:39:09.814 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:39:09.814 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:39:09.814 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:39:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:39:09.814 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:39:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:39:09.814 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:39:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:39:09.814 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:39:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:39:09.814 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:39:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:39:09.814 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:39:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:39:09.814 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:39:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:39:09.814 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:39:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:39:09.814 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:39:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:39:09.814 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:39:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:39:09.814 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:39:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:39:09.814 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:39:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:39:09.814 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:39:09.814 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:39:09.814 139328 ERROR oslo_service.periodic_task >2018-06-25 03:39:39.695 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:39:39.696 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:39:39.705 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:39:39.706 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:39:39.740 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:39:39.740 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:39:39.740 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:39:39.740 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:39:39.740 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:39:39.740 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:39:39.740 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:39:39.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:39:39.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:39:39.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:39:49.743 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:39:49.785 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:39:49.785 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:39:49.785 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:39:49.785 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:39:49.785 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:39:49.785 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:39:49.785 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:39:49.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:39:49.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.081 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:39:49.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:40:09.788 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:40:09.822 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:40:09.822 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:40:09.822 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:40:09.822 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:40:09.822 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:40:09.822 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:40:09.822 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:40:09.824 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:40:09.825 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.119 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:40:09.825 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:40:09.825 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:40:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:40:09.825 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:40:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:40:09.825 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:40:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:40:09.825 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:40:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:40:09.825 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:40:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:40:09.825 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:40:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:40:09.825 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:40:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:40:09.825 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:40:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:40:09.825 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:40:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:40:09.825 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:40:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:40:09.825 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:40:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:40:09.825 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:40:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:40:09.825 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:40:09.825 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:40:09.825 139328 ERROR oslo_service.periodic_task >2018-06-25 03:40:39.696 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:40:39.697 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:41:39.695 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:41:39.696 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:41:39.703 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:41:39.703 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:41:39.737 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:41:39.737 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:41:39.737 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:41:39.737 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:41:39.737 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:41:39.737 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:41:39.737 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:41:39.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:41:39.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:41:39.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:41:49.740 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:41:49.773 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:41:49.773 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:41:49.773 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:41:49.773 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:41:49.773 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:41:49.773 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:41:49.773 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:41:49.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:41:49.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:41:49.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:42:09.776 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:42:09.812 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:42:09.812 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:42:09.812 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:42:09.812 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:42:09.812 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:42:09.812 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:42:09.812 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:42:09.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:42:09.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:42:09.816 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:42:09.816 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:42:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:42:09.816 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:42:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:42:09.816 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:42:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:42:09.816 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:42:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:42:09.816 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:42:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:42:09.816 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:42:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:42:09.816 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:42:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:42:09.816 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:42:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:42:09.816 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:42:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:42:09.816 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:42:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:42:09.816 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:42:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:42:09.816 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:42:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:42:09.816 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:42:09.816 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:42:09.816 139328 ERROR oslo_service.periodic_task >2018-06-25 03:42:39.697 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:42:39.697 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:42:39.704 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:42:39.705 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:42:39.741 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:42:39.741 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:42:39.741 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:42:39.741 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:42:39.741 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:42:39.741 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:42:39.741 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:42:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:42:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:42:39.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:42:49.744 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:42:49.781 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:42:49.781 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:42:49.781 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:42:49.781 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:42:49.781 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:42:49.781 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:42:49.781 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:42:49.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:42:49.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:42:49.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:43:09.785 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:43:09.819 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:43:09.819 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:43:09.819 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:43:09.819 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:43:09.819 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:43:09.819 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:43:09.819 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:43:09.821 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:43:09.821 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:43:09.822 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:43:09.822 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:43:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:43:09.822 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:43:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:43:09.822 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:43:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:43:09.822 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:43:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:43:09.822 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:43:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:43:09.822 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:43:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:43:09.822 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:43:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:43:09.822 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:43:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:43:09.822 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:43:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:43:09.822 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:43:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:43:09.822 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:43:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:43:09.822 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:43:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:43:09.822 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:43:09.822 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:43:09.822 139328 ERROR oslo_service.periodic_task >2018-06-25 03:43:39.693 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:43:39.694 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:43:39.704 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:43:39.704 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:43:39.739 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:43:39.739 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:43:39.739 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:43:39.739 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:43:39.739 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:43:39.739 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:43:39.739 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:43:39.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:43:39.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:43:39.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:43:49.742 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:43:49.776 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:43:49.776 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:43:49.776 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:43:49.776 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:43:49.776 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:43:49.776 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:43:49.776 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:43:49.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:43:49.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:43:49.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:44:09.779 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:44:09.813 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:44:09.813 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:44:09.813 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:44:09.813 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:44:09.813 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:44:09.813 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:44:09.813 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:44:09.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:44:09.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:44:09.816 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:44:09.816 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:44:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:44:09.816 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:44:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:44:09.816 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:44:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:44:09.816 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:44:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:44:09.816 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:44:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:44:09.816 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:44:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:44:09.816 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:44:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:44:09.816 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:44:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:44:09.816 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:44:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:44:09.816 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:44:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:44:09.816 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:44:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:44:09.816 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:44:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:44:09.816 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:44:09.816 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:44:09.816 139328 ERROR oslo_service.periodic_task >2018-06-25 03:44:39.697 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:44:39.697 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:44:39.704 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:44:39.705 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:44:39.739 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:44:39.739 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:44:39.739 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:44:39.739 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:44:39.739 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:44:39.739 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:44:39.739 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:44:39.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:44:39.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:44:39.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:44:49.743 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:44:49.781 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:44:49.781 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:44:49.781 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:44:49.781 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:44:49.781 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:44:49.781 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:44:49.781 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:44:49.783 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:44:49.783 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:44:49.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:45:09.784 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:45:09.820 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:45:09.820 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:45:09.820 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:45:09.820 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:45:09.820 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:45:09.820 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:45:09.820 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:45:09.821 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:45:09.822 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:45:09.822 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:45:09.822 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:45:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:45:09.822 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:45:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:45:09.822 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:45:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:45:09.822 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:45:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:45:09.822 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:45:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:45:09.822 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:45:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:45:09.822 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:45:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:45:09.822 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:45:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:45:09.822 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:45:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:45:09.822 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:45:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:45:09.822 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:45:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:45:09.822 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:45:09.822 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:45:09.822 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:45:09.822 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:45:09.822 139328 ERROR oslo_service.periodic_task >2018-06-25 03:45:39.693 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:45:39.694 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:45:39.703 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:45:39.703 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:45:39.736 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:45:39.736 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:45:39.736 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:45:39.736 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:45:39.736 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:45:39.736 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:45:39.736 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:45:39.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:45:39.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:45:39.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:45:49.739 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:45:49.772 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:45:49.772 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:45:49.772 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:45:49.772 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:45:49.772 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:45:49.772 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:45:49.772 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:45:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:45:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.07 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:45:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:46:09.775 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:46:09.808 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:46:09.808 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:46:09.808 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:46:09.808 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:46:09.808 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:46:09.808 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:46:09.808 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:46:09.810 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:46:09.810 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.106 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:46:09.810 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:46:09.810 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:46:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:46:09.810 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:46:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:46:09.810 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:46:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:46:09.810 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:46:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:46:09.810 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:46:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:46:09.810 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:46:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:46:09.810 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:46:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:46:09.810 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:46:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:46:09.810 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:46:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:46:09.810 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:46:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:46:09.810 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:46:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:46:09.810 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:46:09.810 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:46:09.810 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:46:09.810 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:46:09.810 139328 ERROR oslo_service.periodic_task >2018-06-25 03:46:39.692 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:46:39.693 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:46:39.700 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:46:39.701 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:46:39.738 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:46:39.738 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:46:39.738 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:46:39.738 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:46:39.738 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:46:39.738 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:46:39.738 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:46:39.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:46:39.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:46:39.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:46:49.741 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:46:49.776 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:46:49.776 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:46:49.776 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:46:49.776 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:46:49.776 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:46:49.776 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:46:49.776 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:46:49.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:46:49.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:46:49.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:47:09.779 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:47:09.820 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:47:09.820 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:47:09.820 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:47:09.820 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:47:09.820 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:47:09.820 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:47:09.820 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:47:09.822 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:47:09.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.122 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:47:09.823 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:47:09.823 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:47:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:47:09.823 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:47:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:47:09.823 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:47:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:47:09.823 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:47:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:47:09.823 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:47:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:47:09.823 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:47:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:47:09.823 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:47:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:47:09.823 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:47:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:47:09.823 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:47:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:47:09.823 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:47:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:47:09.823 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:47:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:47:09.823 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:47:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:47:09.823 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:47:09.823 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:47:09.823 139328 ERROR oslo_service.periodic_task >2018-06-25 03:47:39.695 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:47:39.696 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:47:39.702 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:47:39.703 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:47:39.737 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:47:39.737 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:47:39.737 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:47:39.737 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:47:39.737 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:47:39.737 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:47:39.737 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:47:39.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:47:39.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:47:39.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:47:49.740 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:47:49.776 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:47:49.776 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:47:49.776 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:47:49.776 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:47:49.776 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:47:49.776 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:47:49.776 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:47:49.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:47:49.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:47:49.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:48:09.780 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:48:09.818 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:48:09.818 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:48:09.818 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:48:09.818 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:48:09.818 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:48:09.818 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:48:09.818 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:48:09.820 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:48:09.820 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:48:09.821 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:48:09.821 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:48:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:48:09.821 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:48:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:48:09.821 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:48:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:48:09.821 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:48:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:48:09.821 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:48:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:48:09.821 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:48:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:48:09.821 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:48:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:48:09.821 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:48:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:48:09.821 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:48:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:48:09.821 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:48:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:48:09.821 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:48:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:48:09.821 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:48:09.821 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:48:09.821 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:48:09.821 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:48:09.821 139328 ERROR oslo_service.periodic_task >2018-06-25 03:48:39.692 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:48:39.693 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:48:39.710 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:48:39.711 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:48:39.745 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:48:39.745 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:48:39.745 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:48:39.745 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:48:39.745 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:48:39.745 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:48:39.745 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:48:39.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:48:39.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:48:39.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:48:49.748 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:48:49.782 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:48:49.782 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:48:49.782 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:48:49.782 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:48:49.782 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:48:49.782 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:48:49.782 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:48:49.783 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:48:49.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:48:49.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:49:09.785 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:49:09.828 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:49:09.828 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:49:09.828 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:49:09.828 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:49:09.828 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:49:09.828 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:49:09.828 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:49:09.831 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:49:09.831 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.119 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:49:09.831 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:49:09.831 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:49:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:49:09.831 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:49:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:49:09.831 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:49:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:49:09.831 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:49:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:49:09.831 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:49:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:49:09.831 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:49:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:49:09.831 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:49:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:49:09.831 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:49:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:49:09.831 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:49:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:49:09.831 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:49:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:49:09.831 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:49:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:49:09.831 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:49:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:49:09.831 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:49:09.831 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:49:09.831 139328 ERROR oslo_service.periodic_task >2018-06-25 03:49:39.693 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:49:39.693 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:49:39.699 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:49:39.700 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:49:39.733 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:49:39.733 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:49:39.733 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:49:39.733 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:49:39.733 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:49:39.733 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:49:39.733 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:49:39.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:49:39.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:49:39.735 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:49:49.736 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:49:49.770 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:49:49.770 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:49:49.770 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:49:49.770 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:49:49.770 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:49:49.770 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:49:49.770 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:49:49.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:49:49.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:49:49.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:50:09.773 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:50:09.817 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:50:09.817 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:50:09.817 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:50:09.817 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:50:09.817 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:50:09.817 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:50:09.817 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:50:09.819 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:50:09.820 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.12 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:50:09.820 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:50:09.820 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:50:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:50:09.820 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:50:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:50:09.820 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:50:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:50:09.820 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:50:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:50:09.820 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:50:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:50:09.820 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:50:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:50:09.820 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:50:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:50:09.820 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:50:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:50:09.820 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:50:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:50:09.820 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:50:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:50:09.820 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:50:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:50:09.820 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:50:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:50:09.820 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:50:09.820 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:50:09.820 139328 ERROR oslo_service.periodic_task >2018-06-25 03:50:39.692 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:50:39.693 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:50:39.700 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:50:39.701 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:50:39.740 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:50:39.740 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:50:39.740 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:50:39.740 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:50:39.740 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:50:39.740 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:50:39.740 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:50:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:50:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.041 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:50:39.743 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:50:49.744 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:50:49.781 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:50:49.781 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:50:49.781 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:50:49.781 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:50:49.781 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:50:49.781 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:50:49.781 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:50:49.783 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:50:49.783 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.081 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:50:49.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:51:09.784 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:51:09.818 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:51:09.818 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:51:09.818 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:51:09.818 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:51:09.818 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:51:09.818 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:51:09.818 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:51:09.820 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:51:09.820 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.118 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:51:09.820 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:51:09.820 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:51:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:51:09.820 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:51:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:51:09.820 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:51:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:51:09.820 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:51:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:51:09.820 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:51:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:51:09.820 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:51:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:51:09.820 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:51:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:51:09.820 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:51:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:51:09.820 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:51:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:51:09.820 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:51:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:51:09.820 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:51:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:51:09.820 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:51:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:51:09.820 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:51:09.820 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:51:09.820 139328 ERROR oslo_service.periodic_task >2018-06-25 03:51:39.692 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:51:39.693 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:51:39.700 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:51:39.700 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:51:39.735 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:51:39.735 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:51:39.735 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:51:39.735 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:51:39.735 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:51:39.735 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:51:39.735 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:51:39.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:51:39.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:51:39.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:51:49.739 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:51:49.776 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:51:49.776 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:51:49.776 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:51:49.776 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:51:49.776 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:51:49.776 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:51:49.776 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:51:49.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:51:49.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:51:49.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:52:09.780 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:52:09.811 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:52:09.811 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:52:09.811 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:52:09.811 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:52:09.811 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:52:09.811 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:52:09.811 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:52:09.813 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:52:09.814 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:52:09.814 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:52:09.814 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:52:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:52:09.814 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:52:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:52:09.814 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:52:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:52:09.814 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:52:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:52:09.814 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:52:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:52:09.814 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:52:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:52:09.814 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:52:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:52:09.814 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:52:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:52:09.814 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:52:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:52:09.814 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:52:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:52:09.814 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:52:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:52:09.814 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:52:09.814 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:52:09.814 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:52:09.814 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:52:09.814 139328 ERROR oslo_service.periodic_task >2018-06-25 03:52:39.696 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:52:39.696 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:52:39.702 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:52:39.703 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:52:39.736 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:52:39.736 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:52:39.736 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:52:39.736 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:52:39.736 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:52:39.736 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:52:39.736 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:52:39.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:52:39.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:52:39.739 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:52:49.739 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:52:49.772 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:52:49.772 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:52:49.772 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:52:49.772 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:52:49.772 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:52:49.772 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:52:49.772 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:52:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:52:49.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:52:49.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:53:09.775 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:53:09.813 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:53:09.813 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:53:09.813 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:53:09.813 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:53:09.813 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:53:09.813 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:53:09.813 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:53:09.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:53:09.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:53:09.816 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:53:09.816 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:53:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:53:09.816 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:53:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:53:09.816 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:53:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:53:09.816 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:53:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:53:09.816 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:53:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:53:09.816 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:53:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:53:09.816 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:53:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:53:09.816 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:53:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:53:09.816 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:53:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:53:09.816 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:53:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:53:09.816 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:53:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:53:09.816 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:53:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:53:09.816 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:53:09.816 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:53:09.816 139328 ERROR oslo_service.periodic_task >2018-06-25 03:53:39.697 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:53:39.698 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:53:39.705 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:53:39.706 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:53:39.749 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:53:39.749 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:53:39.749 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:53:39.749 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:53:39.749 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:53:39.749 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:53:39.749 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:53:39.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:53:39.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.045 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:53:39.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:53:49.753 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:53:49.786 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:53:49.786 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:53:49.786 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:53:49.786 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:53:49.786 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:53:49.786 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:53:49.786 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:53:49.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:53:49.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.082 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:53:49.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:54:09.790 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:54:09.822 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:54:09.822 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:54:09.822 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:54:09.822 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:54:09.822 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:54:09.822 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:54:09.822 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:54:09.824 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:54:09.824 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:54:09.825 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:54:09.825 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:54:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:54:09.825 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:54:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:54:09.825 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:54:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:54:09.825 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:54:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:54:09.825 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:54:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:54:09.825 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:54:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:54:09.825 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:54:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:54:09.825 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:54:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:54:09.825 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:54:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:54:09.825 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:54:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:54:09.825 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:54:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:54:09.825 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:54:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:54:09.825 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:54:09.825 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:54:09.825 139328 ERROR oslo_service.periodic_task >2018-06-25 03:54:39.696 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:54:39.697 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:54:39.703 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:54:39.704 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:54:39.742 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:54:39.742 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:54:39.742 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:54:39.742 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:54:39.742 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:54:39.742 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:54:39.742 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:54:39.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:54:39.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:54:39.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:54:49.746 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:54:49.787 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:54:49.787 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:54:49.787 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:54:49.787 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:54:49.787 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:54:49.787 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:54:49.787 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:54:49.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:54:49.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.084 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:54:49.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:55:09.790 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:55:09.822 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:55:09.822 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:55:09.822 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:55:09.822 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:55:09.822 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:55:09.822 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:55:09.822 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:55:09.824 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:55:09.824 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.119 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:55:09.825 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:55:09.825 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:55:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:55:09.825 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:55:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:55:09.825 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:55:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:55:09.825 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:55:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:55:09.825 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:55:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:55:09.825 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:55:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:55:09.825 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:55:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:55:09.825 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:55:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:55:09.825 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:55:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:55:09.825 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:55:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:55:09.825 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:55:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:55:09.825 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:55:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:55:09.825 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:55:09.825 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:55:09.825 139328 ERROR oslo_service.periodic_task >2018-06-25 03:55:39.696 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:55:39.697 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:55:39.702 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:55:39.703 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:55:39.737 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:55:39.737 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:55:39.737 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:55:39.737 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:55:39.737 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:55:39.737 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:55:39.737 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:55:39.740 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:55:39.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:55:39.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:55:49.742 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:55:49.779 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:55:49.779 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:55:49.779 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:55:49.779 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:55:49.779 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:55:49.779 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:55:49.779 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:55:49.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:55:49.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:55:49.783 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:56:09.783 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:56:09.816 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:56:09.816 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:56:09.816 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:56:09.816 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:56:09.816 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:56:09.816 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:56:09.816 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:56:09.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:56:09.818 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:56:09.818 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:56:09.818 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:56:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:56:09.818 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:56:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:56:09.818 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:56:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:56:09.818 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:56:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:56:09.818 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:56:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:56:09.818 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:56:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:56:09.818 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:56:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:56:09.818 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:56:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:56:09.818 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:56:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:56:09.818 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:56:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:56:09.818 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:56:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:56:09.818 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:56:09.818 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:56:09.818 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:56:09.818 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:56:09.818 139328 ERROR oslo_service.periodic_task >2018-06-25 03:56:39.699 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:56:39.700 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:56:39.707 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:56:39.708 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:56:39.749 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:56:39.749 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:56:39.749 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:56:39.749 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:56:39.749 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:56:39.749 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:56:39.749 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:56:39.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:56:39.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.043 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:56:39.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:56:49.752 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:56:49.791 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:56:49.791 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:56:49.791 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:56:49.791 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:56:49.791 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:56:49.791 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:56:49.791 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:56:49.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:56:49.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.086 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:56:49.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:57:09.794 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:57:09.829 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:57:09.829 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:57:09.829 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:57:09.829 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:57:09.829 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:57:09.829 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:57:09.829 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:57:09.831 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:57:09.831 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.124 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:57:09.832 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:57:09.832 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:57:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:57:09.832 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:57:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:57:09.832 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:57:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:57:09.832 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:57:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:57:09.832 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:57:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:57:09.832 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:57:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:57:09.832 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:57:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:57:09.832 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:57:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:57:09.832 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:57:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:57:09.832 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:57:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:57:09.832 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:57:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:57:09.832 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:57:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:57:09.832 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:57:09.832 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:57:09.832 139328 ERROR oslo_service.periodic_task >2018-06-25 03:57:39.703 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:57:39.704 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:57:39.710 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:57:39.711 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:57:39.745 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:57:39.745 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:57:39.745 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:57:39.745 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:57:39.745 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:57:39.745 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:57:39.745 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:57:39.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:57:39.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:57:39.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:57:49.748 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:57:49.786 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:57:49.786 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:57:49.786 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:57:49.786 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:57:49.786 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:57:49.786 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:57:49.786 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:57:49.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:57:49.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:57:49.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:58:09.789 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:58:09.826 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:58:09.826 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:58:09.826 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:58:09.826 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:58:09.826 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:58:09.826 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:58:09.826 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:58:09.829 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:58:09.829 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.119 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:58:09.830 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:58:09.830 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:58:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:58:09.830 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:58:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:58:09.830 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:58:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:58:09.830 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:58:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:58:09.830 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:58:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:58:09.830 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:58:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:58:09.830 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:58:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:58:09.830 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:58:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:58:09.830 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:58:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:58:09.830 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:58:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:58:09.830 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:58:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:58:09.830 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:58:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:58:09.830 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:58:09.830 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:58:09.830 139328 ERROR oslo_service.periodic_task >2018-06-25 03:58:39.701 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:58:39.702 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:58:39.708 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:58:39.709 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:58:39.743 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:58:39.743 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:58:39.743 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:58:39.743 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:58:39.743 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:58:39.743 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:58:39.743 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:58:39.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:58:39.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:58:39.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:58:49.746 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:58:49.781 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:58:49.781 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:58:49.781 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:58:49.781 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:58:49.781 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:58:49.781 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:58:49.781 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:58:49.783 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:58:49.783 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:58:49.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:59:09.784 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:59:09.816 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:59:09.816 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:59:09.816 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:59:09.816 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:59:09.816 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:59:09.816 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:59:09.816 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:59:09.819 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:59:09.819 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:59:09.820 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:59:09.820 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 03:59:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 03:59:09.820 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 03:59:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 03:59:09.820 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 03:59:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 03:59:09.820 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 03:59:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 03:59:09.820 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 03:59:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 03:59:09.820 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 03:59:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 03:59:09.820 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 03:59:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 03:59:09.820 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 03:59:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 03:59:09.820 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 03:59:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 03:59:09.820 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 03:59:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 03:59:09.820 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 03:59:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 03:59:09.820 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 03:59:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 03:59:09.820 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 03:59:09.820 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 03:59:09.820 139328 ERROR oslo_service.periodic_task >2018-06-25 03:59:39.701 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:59:39.702 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 03:59:39.711 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 03:59:39.712 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:59:39.747 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:59:39.747 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:59:39.747 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:59:39.747 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:59:39.747 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:59:39.747 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:59:39.747 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:59:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:59:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:59:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 03:59:49.751 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 03:59:49.786 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 03:59:49.786 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 03:59:49.786 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 03:59:49.786 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 03:59:49.786 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 03:59:49.786 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 03:59:49.786 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 03:59:49.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 03:59:49.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 03:59:49.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:00:09.790 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:00:09.826 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:00:09.826 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:00:09.826 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:00:09.826 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:00:09.826 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:00:09.826 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:00:09.826 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:00:09.828 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:00:09.828 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:00:09.829 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:00:09.829 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:00:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:00:09.829 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:00:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:00:09.829 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:00:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:00:09.829 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:00:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:00:09.829 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:00:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:00:09.829 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:00:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:00:09.829 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:00:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:00:09.829 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:00:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:00:09.829 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:00:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:00:09.829 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:00:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:00:09.829 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:00:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:00:09.829 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:00:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:00:09.829 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:00:09.829 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:00:09.829 139328 ERROR oslo_service.periodic_task >2018-06-25 04:00:39.700 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:00:39.701 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:00:39.735 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:00:39.735 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:00:39.735 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:00:39.735 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:00:39.735 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:00:39.735 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:00:39.735 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:00:39.737 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:00:39.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:00:39.738 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:00:49.739 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:00:49.773 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:00:49.773 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:00:49.773 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:00:49.773 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:00:49.773 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:00:49.773 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:00:49.773 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:00:49.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:00:49.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:00:49.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:01:09.776 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:01:09.813 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:01:09.813 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:01:09.813 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:01:09.813 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:01:09.813 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:01:09.813 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:01:09.813 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:01:09.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:01:09.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:01:09.817 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:01:09.817 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:01:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:01:09.817 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:01:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:01:09.817 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:01:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:01:09.817 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:01:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:01:09.817 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:01:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:01:09.817 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:01:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:01:09.817 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:01:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:01:09.817 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:01:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:01:09.817 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:01:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:01:09.817 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:01:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:01:09.817 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:01:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:01:09.817 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:01:09.817 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:01:09.817 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:01:09.817 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:01:09.817 139328 ERROR oslo_service.periodic_task >2018-06-25 04:01:39.698 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:01:39.699 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:01:39.706 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:01:39.706 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:01:39.739 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:01:39.739 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:01:39.739 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:01:39.739 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:01:39.739 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:01:39.739 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:01:39.739 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:01:39.741 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:01:39.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:01:39.742 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:01:49.743 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:01:49.777 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:01:49.777 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:01:49.777 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:01:49.777 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:01:49.777 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:01:49.777 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:01:49.777 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:01:49.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:01:49.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:01:49.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:02:09.780 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:02:09.813 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:02:09.813 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:02:09.813 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:02:09.813 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:02:09.813 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:02:09.813 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:02:09.813 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:02:09.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:02:09.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:02:09.816 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:02:09.816 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:02:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:02:09.816 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:02:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:02:09.816 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:02:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:02:09.816 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:02:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:02:09.816 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:02:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:02:09.816 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:02:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:02:09.816 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:02:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:02:09.816 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:02:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:02:09.816 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:02:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:02:09.816 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:02:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:02:09.816 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:02:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:02:09.816 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:02:09.816 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:02:09.816 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:02:09.816 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:02:09.816 139328 ERROR oslo_service.periodic_task >2018-06-25 04:02:39.698 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:02:39.698 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:02:39.707 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:02:39.708 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:02:39.743 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:02:39.743 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:02:39.743 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:02:39.743 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:02:39.743 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:02:39.743 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:02:39.743 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:02:39.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:02:39.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:02:39.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:02:49.747 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:02:49.787 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:02:49.787 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:02:49.787 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:02:49.787 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:02:49.787 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:02:49.787 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:02:49.787 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:02:49.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:02:49.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.081 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:02:49.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:03:09.791 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:03:09.838 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:03:09.838 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:03:09.838 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:03:09.838 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:03:09.838 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:03:09.838 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:03:09.838 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:03:09.840 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:03:09.840 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.131 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:03:09.841 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:03:09.841 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:03:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:03:09.841 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:03:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:03:09.841 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:03:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:03:09.841 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:03:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:03:09.841 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:03:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:03:09.841 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:03:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:03:09.841 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:03:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:03:09.841 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:03:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:03:09.841 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:03:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:03:09.841 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:03:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:03:09.841 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:03:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:03:09.841 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:03:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:03:09.841 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:03:09.841 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:03:09.841 139328 ERROR oslo_service.periodic_task >2018-06-25 04:03:39.702 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:03:39.703 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:03:39.711 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:03:39.711 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:03:39.748 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:03:39.748 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:03:39.748 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:03:39.748 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:03:39.748 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:03:39.748 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:03:39.748 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:03:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:03:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:03:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:03:49.751 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:03:49.787 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:03:49.787 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:03:49.787 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:03:49.787 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:03:49.787 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:03:49.787 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:03:49.787 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:03:49.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:03:49.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:03:49.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:04:09.791 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:04:09.825 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:04:09.825 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:04:09.825 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:04:09.825 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:04:09.825 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:04:09.825 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:04:09.825 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:04:09.828 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:04:09.828 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:04:09.828 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:04:09.828 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:04:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:04:09.828 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:04:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:04:09.828 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:04:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:04:09.828 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:04:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:04:09.828 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:04:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:04:09.828 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:04:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:04:09.828 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:04:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:04:09.828 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:04:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:04:09.828 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:04:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:04:09.828 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:04:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:04:09.828 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:04:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:04:09.828 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:04:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:04:09.828 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:04:09.828 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:04:09.828 139328 ERROR oslo_service.periodic_task >2018-06-25 04:04:39.700 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:04:39.701 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:04:39.707 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:04:39.708 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:04:39.742 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:04:39.742 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:04:39.742 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:04:39.742 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:04:39.742 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:04:39.742 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:04:39.742 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:04:39.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:04:39.744 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:04:39.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:04:49.745 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:04:49.782 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:04:49.782 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:04:49.782 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:04:49.782 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:04:49.782 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:04:49.782 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:04:49.782 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:04:49.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:04:49.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:04:49.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:05:09.785 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:05:09.817 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:05:09.817 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:05:09.817 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:05:09.817 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:05:09.817 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:05:09.817 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:05:09.817 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:05:09.819 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:05:09.819 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:05:09.820 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:05:09.820 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:05:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:05:09.820 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:05:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:05:09.820 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:05:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:05:09.820 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:05:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:05:09.820 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:05:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:05:09.820 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:05:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:05:09.820 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:05:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:05:09.820 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:05:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:05:09.820 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:05:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:05:09.820 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:05:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:05:09.820 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:05:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:05:09.820 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:05:09.820 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:05:09.820 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:05:09.820 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:05:09.820 139328 ERROR oslo_service.periodic_task >2018-06-25 04:05:39.701 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:05:39.702 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:05:39.708 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:05:39.708 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:05:39.743 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:05:39.743 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:05:39.743 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:05:39.743 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:05:39.743 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:05:39.743 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:05:39.743 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:05:39.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:05:39.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:05:39.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:05:49.746 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:05:49.781 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:05:49.781 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:05:49.781 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:05:49.781 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:05:49.781 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:05:49.781 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:05:49.781 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:05:49.783 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:05:49.783 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:05:49.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:06:09.784 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:06:09.821 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:06:09.821 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:06:09.821 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:06:09.821 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:06:09.821 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:06:09.821 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:06:09.821 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:06:09.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:06:09.824 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:06:09.824 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:06:09.824 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:06:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:06:09.824 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:06:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:06:09.824 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:06:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:06:09.824 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:06:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:06:09.824 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:06:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:06:09.824 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:06:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:06:09.824 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:06:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:06:09.824 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:06:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:06:09.824 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:06:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:06:09.824 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:06:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:06:09.824 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:06:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:06:09.824 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:06:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:06:09.824 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:06:09.824 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:06:09.824 139328 ERROR oslo_service.periodic_task >2018-06-25 04:06:39.705 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:06:39.706 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:06:39.713 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:06:39.713 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:06:39.745 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:06:39.745 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:06:39.745 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:06:39.745 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:06:39.745 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:06:39.745 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:06:39.745 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:06:39.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:06:39.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:06:39.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:06:49.748 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:06:49.783 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:06:49.783 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:06:49.783 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:06:49.783 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:06:49.783 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:06:49.783 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:06:49.783 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:06:49.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:06:49.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:06:49.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:07:09.786 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:07:09.821 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:07:09.821 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:07:09.821 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:07:09.821 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:07:09.821 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:07:09.821 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:07:09.821 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:07:09.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:07:09.824 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:07:09.824 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:07:09.824 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:07:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:07:09.824 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:07:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:07:09.824 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:07:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:07:09.824 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:07:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:07:09.824 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:07:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:07:09.824 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:07:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:07:09.824 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:07:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:07:09.824 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:07:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:07:09.824 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:07:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:07:09.824 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:07:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:07:09.824 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:07:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:07:09.824 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:07:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:07:09.824 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:07:09.824 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:07:09.824 139328 ERROR oslo_service.periodic_task >2018-06-25 04:07:39.705 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:07:39.706 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:07:39.713 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:07:39.713 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:07:39.747 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:07:39.747 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:07:39.747 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:07:39.747 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:07:39.747 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:07:39.747 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:07:39.747 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:07:39.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:07:39.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:07:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:07:49.750 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:07:49.790 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:07:49.790 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:07:49.790 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:07:49.790 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:07:49.790 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:07:49.790 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:07:49.790 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:07:49.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:07:49.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:07:49.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:08:09.794 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:08:09.830 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:08:09.830 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:08:09.830 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:08:09.830 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:08:09.830 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:08:09.830 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:08:09.830 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:08:09.833 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:08:09.833 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.119 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:08:09.833 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:08:09.833 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:08:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:08:09.833 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:08:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:08:09.833 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:08:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:08:09.833 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:08:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:08:09.833 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:08:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:08:09.833 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:08:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:08:09.833 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:08:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:08:09.833 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:08:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:08:09.833 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:08:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:08:09.833 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:08:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:08:09.833 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:08:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:08:09.833 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:08:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:08:09.833 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:08:09.833 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:08:09.833 139328 ERROR oslo_service.periodic_task >2018-06-25 04:08:39.704 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:08:39.705 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:08:39.711 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:08:39.712 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:08:39.745 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:08:39.745 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:08:39.745 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:08:39.745 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:08:39.745 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:08:39.745 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:08:39.745 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:08:39.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:08:39.747 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:08:39.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:08:49.748 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:08:49.786 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:08:49.786 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:08:49.786 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:08:49.786 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:08:49.786 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:08:49.786 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:08:49.786 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:08:49.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:08:49.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:08:49.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:09:09.789 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:09:09.831 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:09:09.831 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:09:09.831 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:09:09.831 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:09:09.831 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:09:09.831 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:09:09.831 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:09:09.833 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:09:09.833 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.121 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:09:09.834 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:09:09.834 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:09:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:09:09.834 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:09:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:09:09.834 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:09:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:09:09.834 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:09:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:09:09.834 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:09:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:09:09.834 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:09:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:09:09.834 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:09:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:09:09.834 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:09:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:09:09.834 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:09:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:09:09.834 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:09:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:09:09.834 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:09:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:09:09.834 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:09:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:09:09.834 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:09:09.834 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:09:09.834 139328 ERROR oslo_service.periodic_task >2018-06-25 04:09:39.706 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:09:39.707 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:09:39.714 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:09:39.715 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:09:39.747 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:09:39.747 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:09:39.747 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:09:39.747 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:09:39.747 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:09:39.747 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:09:39.747 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:09:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:09:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:09:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:09:49.751 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:09:49.784 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:09:49.784 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:09:49.784 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:09:49.784 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:09:49.784 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:09:49.784 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:09:49.784 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:09:49.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:09:49.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:09:49.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:10:09.788 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:10:09.821 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:10:09.821 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:10:09.821 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:10:09.821 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:10:09.821 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:10:09.821 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:10:09.821 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:10:09.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:10:09.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:10:09.824 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:10:09.824 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:10:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:10:09.824 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:10:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:10:09.824 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:10:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:10:09.824 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:10:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:10:09.824 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:10:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:10:09.824 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:10:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:10:09.824 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:10:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:10:09.824 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:10:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:10:09.824 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:10:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:10:09.824 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:10:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:10:09.824 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:10:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:10:09.824 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:10:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:10:09.824 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:10:09.824 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:10:09.824 139328 ERROR oslo_service.periodic_task >2018-06-25 04:10:39.705 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:10:39.706 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:10:39.713 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:10:39.713 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:10:39.756 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:10:39.756 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:10:39.756 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:10:39.756 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:10:39.756 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:10:39.756 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:10:39.756 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:10:39.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:10:39.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.045 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:10:39.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:10:49.759 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:10:49.792 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:10:49.792 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:10:49.792 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:10:49.792 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:10:49.792 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:10:49.792 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:10:49.792 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:10:49.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:10:49.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.08 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:10:49.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:11:09.795 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:11:09.831 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:11:09.831 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:11:09.831 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:11:09.831 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:11:09.831 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:11:09.831 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:11:09.831 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:11:09.833 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:11:09.834 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.12 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:11:09.834 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:11:09.834 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:11:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:11:09.834 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:11:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:11:09.834 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:11:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:11:09.834 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:11:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:11:09.834 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:11:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:11:09.834 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:11:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:11:09.834 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:11:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:11:09.834 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:11:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:11:09.834 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:11:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:11:09.834 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:11:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:11:09.834 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:11:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:11:09.834 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:11:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:11:09.834 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:11:09.834 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:11:09.834 139328 ERROR oslo_service.periodic_task >2018-06-25 04:11:39.705 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:11:39.706 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:11:39.713 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:11:39.713 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:11:39.747 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:11:39.747 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:11:39.747 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:11:39.747 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:11:39.747 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:11:39.747 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:11:39.747 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:11:39.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:11:39.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:11:39.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:11:49.750 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:11:49.784 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:11:49.784 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:11:49.784 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:11:49.784 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:11:49.784 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:11:49.784 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:11:49.784 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:11:49.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:11:49.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:11:49.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:12:09.787 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:12:09.821 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:12:09.821 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:12:09.821 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:12:09.821 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:12:09.821 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:12:09.821 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:12:09.821 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:12:09.824 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:12:09.824 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:12:09.824 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:12:09.824 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:12:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:12:09.824 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:12:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:12:09.824 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:12:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:12:09.824 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:12:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:12:09.824 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:12:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:12:09.824 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:12:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:12:09.824 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:12:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:12:09.824 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:12:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:12:09.824 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:12:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:12:09.824 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:12:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:12:09.824 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:12:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:12:09.824 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:12:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:12:09.824 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:12:09.824 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:12:09.824 139328 ERROR oslo_service.periodic_task >2018-06-25 04:12:39.706 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:12:39.707 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:12:39.716 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:12:39.716 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:12:39.756 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:12:39.756 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:12:39.756 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:12:39.756 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:12:39.756 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:12:39.756 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:12:39.756 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:12:39.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:12:39.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.044 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:12:39.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:12:49.762 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:12:49.797 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:12:49.797 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:12:49.797 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:12:49.797 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:12:49.797 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:12:49.797 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:12:49.797 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:12:49.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:12:49.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.083 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:12:49.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:13:09.801 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:13:09.836 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:13:09.836 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:13:09.836 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:13:09.836 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:13:09.836 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:13:09.836 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:13:09.836 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:13:09.839 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:13:09.839 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.122 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:13:09.840 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:13:09.840 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:13:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:13:09.840 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:13:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:13:09.840 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:13:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:13:09.840 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:13:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:13:09.840 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:13:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:13:09.840 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:13:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:13:09.840 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:13:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:13:09.840 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:13:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:13:09.840 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:13:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:13:09.840 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:13:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:13:09.840 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:13:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:13:09.840 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:13:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:13:09.840 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:13:09.840 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:13:09.840 139328 ERROR oslo_service.periodic_task >2018-06-25 04:13:39.701 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:13:39.702 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:13:39.709 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:13:39.709 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:13:39.743 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:13:39.743 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:13:39.743 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:13:39.743 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:13:39.743 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:13:39.743 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:13:39.743 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:13:39.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:13:39.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:13:39.746 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:13:49.747 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:13:49.781 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:13:49.781 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:13:49.781 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:13:49.781 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:13:49.781 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:13:49.781 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:13:49.781 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:13:49.783 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:13:49.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:13:49.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:14:09.785 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:14:09.826 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:14:09.826 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:14:09.826 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:14:09.826 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:14:09.826 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:14:09.826 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:14:09.826 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:14:09.828 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:14:09.828 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.118 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:14:09.828 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:14:09.828 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:14:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:14:09.828 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:14:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:14:09.828 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:14:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:14:09.828 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:14:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:14:09.828 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:14:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:14:09.828 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:14:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:14:09.828 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:14:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:14:09.828 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:14:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:14:09.828 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:14:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:14:09.828 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:14:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:14:09.828 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:14:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:14:09.828 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:14:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:14:09.828 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:14:09.828 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:14:09.828 139328 ERROR oslo_service.periodic_task >2018-06-25 04:14:39.700 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:14:39.700 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:14:39.711 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:14:39.712 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:14:39.743 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:14:39.743 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:14:39.743 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:14:39.743 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:14:39.743 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:14:39.743 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:14:39.743 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:14:39.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:14:39.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.033 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:14:39.745 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:14:49.746 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:14:49.782 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:14:49.782 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:14:49.782 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:14:49.782 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:14:49.782 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:14:49.782 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:14:49.782 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:14:49.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:14:49.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:14:49.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:15:09.785 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:15:09.819 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:15:09.819 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:15:09.819 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:15:09.819 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:15:09.819 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:15:09.819 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:15:09.819 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:15:09.822 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:15:09.822 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:15:09.823 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:15:09.823 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:15:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:15:09.823 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:15:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:15:09.823 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:15:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:15:09.823 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:15:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:15:09.823 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:15:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:15:09.823 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:15:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:15:09.823 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:15:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:15:09.823 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:15:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:15:09.823 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:15:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:15:09.823 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:15:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:15:09.823 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:15:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:15:09.823 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:15:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:15:09.823 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:15:09.823 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:15:09.823 139328 ERROR oslo_service.periodic_task >2018-06-25 04:15:39.704 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:15:39.705 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:15:39.714 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:15:39.714 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:15:39.749 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:15:39.749 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:15:39.749 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:15:39.749 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:15:39.749 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:15:39.749 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:15:39.749 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:15:39.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:15:39.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:15:39.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:15:49.752 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:15:49.785 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:15:49.785 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:15:49.785 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:15:49.785 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:15:49.785 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:15:49.785 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:15:49.785 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:15:49.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:15:49.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:15:49.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:16:09.788 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:16:09.821 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:16:09.821 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:16:09.821 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:16:09.821 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:16:09.821 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:16:09.821 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:16:09.821 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:16:09.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:16:09.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:16:09.823 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:16:09.823 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:16:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:16:09.823 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:16:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:16:09.823 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:16:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:16:09.823 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:16:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:16:09.823 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:16:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:16:09.823 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:16:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:16:09.823 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:16:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:16:09.823 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:16:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:16:09.823 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:16:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:16:09.823 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:16:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:16:09.823 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:16:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:16:09.823 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:16:09.823 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:16:09.823 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:16:09.823 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:16:09.823 139328 ERROR oslo_service.periodic_task >2018-06-25 04:16:39.705 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:16:39.705 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:16:39.712 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:16:39.713 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:16:39.747 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:16:39.747 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:16:39.747 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:16:39.747 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:16:39.747 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:16:39.747 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:16:39.747 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:16:39.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:16:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:16:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:16:49.750 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:16:49.787 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:16:49.787 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:16:49.787 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:16:49.787 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:16:49.787 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:16:49.787 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:16:49.787 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:16:49.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:16:49.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:16:49.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:17:09.791 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:17:09.824 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:17:09.824 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:17:09.824 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:17:09.824 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:17:09.824 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:17:09.824 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:17:09.824 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:17:09.827 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:17:09.827 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:17:09.827 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:17:09.827 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:17:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:17:09.827 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:17:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:17:09.827 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:17:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:17:09.827 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:17:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:17:09.827 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:17:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:17:09.827 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:17:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:17:09.827 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:17:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:17:09.827 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:17:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:17:09.827 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:17:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:17:09.827 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:17:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:17:09.827 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:17:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:17:09.827 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:17:09.827 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:17:09.827 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:17:09.827 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:17:09.827 139328 ERROR oslo_service.periodic_task >2018-06-25 04:17:39.709 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:17:39.710 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:17:39.716 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:17:39.717 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:17:39.754 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:17:39.754 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:17:39.754 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:17:39.754 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:17:39.754 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:17:39.754 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:17:39.754 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:17:39.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:17:39.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:17:39.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:17:49.757 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:17:49.790 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:17:49.790 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:17:49.790 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:17:49.790 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:17:49.790 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:17:49.790 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:17:49.790 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:17:49.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:17:49.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:17:49.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:18:09.793 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:18:09.826 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:18:09.826 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:18:09.826 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:18:09.826 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:18:09.826 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:18:09.826 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:18:09.826 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:18:09.828 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:18:09.828 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:18:09.829 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:18:09.829 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:18:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:18:09.829 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:18:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:18:09.829 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:18:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:18:09.829 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:18:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:18:09.829 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:18:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:18:09.829 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:18:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:18:09.829 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:18:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:18:09.829 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:18:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:18:09.829 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:18:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:18:09.829 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:18:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:18:09.829 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:18:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:18:09.829 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:18:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:18:09.829 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:18:09.829 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:18:09.829 139328 ERROR oslo_service.periodic_task >2018-06-25 04:18:39.710 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:18:39.711 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:18:39.719 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:18:39.720 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:18:39.752 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:18:39.752 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:18:39.752 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:18:39.752 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:18:39.752 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:18:39.752 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:18:39.752 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:18:39.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:18:39.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:18:39.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:18:49.759 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:18:49.794 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:18:49.794 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:18:49.794 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:18:49.794 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:18:49.794 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:18:49.794 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:18:49.794 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:18:49.797 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:18:49.797 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:18:49.798 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:19:09.798 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:19:09.833 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:19:09.833 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:19:09.833 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:19:09.833 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:19:09.833 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:19:09.833 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:19:09.833 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:19:09.835 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:19:09.835 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:19:09.836 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:19:09.836 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:19:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:19:09.836 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:19:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:19:09.836 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:19:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:19:09.836 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:19:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:19:09.836 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:19:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:19:09.836 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:19:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:19:09.836 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:19:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:19:09.836 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:19:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:19:09.836 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:19:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:19:09.836 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:19:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:19:09.836 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:19:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:19:09.836 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:19:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:19:09.836 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:19:09.836 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:19:09.836 139328 ERROR oslo_service.periodic_task >2018-06-25 04:19:39.707 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:19:39.708 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:19:39.716 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:19:39.716 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:19:39.751 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:19:39.751 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:19:39.751 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:19:39.751 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:19:39.751 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:19:39.751 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:19:39.751 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:19:39.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:19:39.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:19:39.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:19:49.754 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:19:49.793 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:19:49.793 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:19:49.793 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:19:49.793 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:19:49.793 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:19:49.793 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:19:49.793 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:19:49.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:19:49.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:19:49.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:20:09.796 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:20:09.831 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:20:09.831 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:20:09.831 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:20:09.831 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:20:09.831 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:20:09.831 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:20:09.831 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:20:09.833 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:20:09.833 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:20:09.834 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:20:09.834 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:20:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:20:09.834 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:20:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:20:09.834 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:20:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:20:09.834 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:20:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:20:09.834 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:20:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:20:09.834 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:20:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:20:09.834 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:20:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:20:09.834 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:20:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:20:09.834 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:20:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:20:09.834 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:20:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:20:09.834 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:20:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:20:09.834 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:20:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:20:09.834 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:20:09.834 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:20:09.834 139328 ERROR oslo_service.periodic_task >2018-06-25 04:20:39.705 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:20:39.706 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:20:39.712 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:20:39.712 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:20:39.746 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:20:39.746 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:20:39.746 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:20:39.746 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:20:39.746 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:20:39.746 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:20:39.746 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:20:39.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:20:39.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:20:39.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:20:49.749 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:20:49.784 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:20:49.784 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:20:49.784 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:20:49.784 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:20:49.784 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:20:49.784 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:20:49.784 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:20:49.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:20:49.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:20:49.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:21:09.788 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:21:09.822 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:21:09.822 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:21:09.822 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:21:09.822 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:21:09.822 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:21:09.822 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:21:09.822 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:21:09.824 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:21:09.824 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:21:09.824 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:21:09.824 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:21:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:21:09.824 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:21:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:21:09.824 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:21:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:21:09.824 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:21:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:21:09.824 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:21:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:21:09.824 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:21:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:21:09.824 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:21:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:21:09.824 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:21:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:21:09.824 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:21:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:21:09.824 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:21:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:21:09.824 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:21:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:21:09.824 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:21:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:21:09.824 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:21:09.824 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:21:09.824 139328 ERROR oslo_service.periodic_task >2018-06-25 04:21:39.705 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:21:39.706 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:21:39.713 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:21:39.713 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:21:39.746 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:21:39.746 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:21:39.746 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:21:39.746 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:21:39.746 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:21:39.746 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:21:39.746 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:21:39.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:21:39.748 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:21:39.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:21:49.749 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:21:49.784 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:21:49.784 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:21:49.784 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:21:49.784 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:21:49.784 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:21:49.784 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:21:49.784 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:21:49.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:21:49.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:21:49.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:22:09.787 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:22:09.821 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:22:09.821 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:22:09.821 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:22:09.821 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:22:09.821 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:22:09.821 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:22:09.821 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:22:09.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:22:09.824 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:22:09.824 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:22:09.824 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:22:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:22:09.824 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:22:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:22:09.824 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:22:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:22:09.824 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:22:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:22:09.824 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:22:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:22:09.824 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:22:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:22:09.824 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:22:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:22:09.824 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:22:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:22:09.824 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:22:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:22:09.824 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:22:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:22:09.824 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:22:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:22:09.824 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:22:09.824 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:22:09.824 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:22:09.824 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:22:09.824 139328 ERROR oslo_service.periodic_task >2018-06-25 04:22:39.706 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:22:39.707 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:22:39.713 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:22:39.714 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:22:39.747 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:22:39.747 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:22:39.747 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:22:39.747 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:22:39.747 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:22:39.747 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:22:39.747 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:22:39.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:22:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:22:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:22:49.750 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:22:49.791 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:22:49.791 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:22:49.791 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:22:49.791 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:22:49.791 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:22:49.791 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:22:49.791 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:22:49.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:22:49.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.08 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:22:49.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:23:09.796 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:23:09.836 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:23:09.836 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:23:09.836 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:23:09.836 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:23:09.836 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:23:09.836 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:23:09.836 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:23:09.838 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:23:09.838 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.124 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:23:09.838 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:23:09.838 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:23:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:23:09.838 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:23:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:23:09.838 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:23:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:23:09.838 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:23:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:23:09.838 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:23:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:23:09.838 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:23:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:23:09.838 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:23:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:23:09.838 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:23:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:23:09.838 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:23:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:23:09.838 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:23:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:23:09.838 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:23:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:23:09.838 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:23:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:23:09.838 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:23:09.838 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:23:09.838 139328 ERROR oslo_service.periodic_task >2018-06-25 04:23:39.710 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:23:39.710 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:23:39.716 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:23:39.717 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:23:39.748 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:23:39.748 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:23:39.748 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:23:39.748 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:23:39.748 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:23:39.748 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:23:39.748 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:23:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:23:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.032 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:23:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:23:49.751 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:23:49.791 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:23:49.791 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:23:49.791 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:23:49.791 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:23:49.791 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:23:49.791 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:23:49.791 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:23:49.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:23:49.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:23:49.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:24:09.795 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:24:09.829 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:24:09.829 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:24:09.829 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:24:09.829 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:24:09.829 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:24:09.829 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:24:09.829 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:24:09.832 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:24:09.832 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:24:09.832 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:24:09.832 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:24:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:24:09.832 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:24:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:24:09.832 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:24:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:24:09.832 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:24:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:24:09.832 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:24:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:24:09.832 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:24:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:24:09.832 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:24:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:24:09.832 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:24:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:24:09.832 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:24:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:24:09.832 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:24:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:24:09.832 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:24:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:24:09.832 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:24:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:24:09.832 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:24:09.832 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:24:09.832 139328 ERROR oslo_service.periodic_task >2018-06-25 04:24:39.714 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:24:39.714 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:24:39.722 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:24:39.722 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:24:39.754 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:24:39.754 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:24:39.754 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:24:39.754 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:24:39.754 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:24:39.754 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:24:39.754 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:24:39.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:24:39.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:24:39.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:24:49.758 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:24:49.794 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:24:49.794 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:24:49.794 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:24:49.794 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:24:49.794 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:24:49.794 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:24:49.794 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:24:49.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:24:49.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:24:49.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:25:09.797 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:25:09.831 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:25:09.831 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:25:09.831 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:25:09.831 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:25:09.831 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:25:09.831 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:25:09.831 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:25:09.834 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:25:09.834 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:25:09.835 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:25:09.835 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:25:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:25:09.835 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:25:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:25:09.835 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:25:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:25:09.835 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:25:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:25:09.835 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:25:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:25:09.835 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:25:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:25:09.835 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:25:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:25:09.835 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:25:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:25:09.835 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:25:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:25:09.835 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:25:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:25:09.835 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:25:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:25:09.835 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:25:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:25:09.835 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:25:09.835 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:25:09.835 139328 ERROR oslo_service.periodic_task >2018-06-25 04:25:39.716 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:25:39.717 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:25:39.730 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:25:39.730 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:25:39.764 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:25:39.764 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:25:39.764 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:25:39.764 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:25:39.764 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:25:39.764 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:25:39.764 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:25:39.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:25:39.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:25:39.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:25:49.768 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:25:49.802 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:25:49.802 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:25:49.802 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:25:49.802 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:25:49.802 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:25:49.802 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:25:49.802 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:25:49.804 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:25:49.804 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:25:49.805 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:26:09.805 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:26:09.839 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:26:09.839 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:26:09.839 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:26:09.839 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:26:09.839 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:26:09.839 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:26:09.839 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:26:09.842 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:26:09.842 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:26:09.843 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:26:09.843 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:26:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:26:09.843 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:26:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:26:09.843 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:26:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:26:09.843 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:26:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:26:09.843 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:26:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:26:09.843 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:26:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:26:09.843 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:26:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:26:09.843 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:26:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:26:09.843 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:26:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:26:09.843 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:26:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:26:09.843 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:26:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:26:09.843 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:26:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:26:09.843 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:26:09.843 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:26:09.843 139328 ERROR oslo_service.periodic_task >2018-06-25 04:26:39.715 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:26:39.716 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:26:39.722 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:26:39.723 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:26:39.757 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:26:39.757 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:26:39.757 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:26:39.757 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:26:39.757 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:26:39.757 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:26:39.757 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:26:39.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:26:39.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:26:39.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:26:49.761 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:26:49.800 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:26:49.800 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:26:49.800 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:26:49.800 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:26:49.800 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:26:49.800 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:26:49.800 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:26:49.803 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:26:49.803 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.08 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:26:49.804 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:27:09.804 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:27:09.838 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:27:09.838 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:27:09.838 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:27:09.838 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:27:09.838 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:27:09.838 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:27:09.838 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:27:09.841 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:27:09.841 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.118 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:27:09.841 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:27:09.841 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:27:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:27:09.841 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:27:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:27:09.841 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:27:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:27:09.841 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:27:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:27:09.841 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:27:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:27:09.841 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:27:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:27:09.841 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:27:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:27:09.841 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:27:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:27:09.841 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:27:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:27:09.841 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:27:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:27:09.841 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:27:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:27:09.841 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:27:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:27:09.841 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:27:09.841 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:27:09.841 139328 ERROR oslo_service.periodic_task >2018-06-25 04:27:39.713 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:27:39.713 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:27:39.720 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:27:39.720 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:27:39.753 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:27:39.753 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:27:39.753 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:27:39.753 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:27:39.753 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:27:39.753 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:27:39.753 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:27:39.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:27:39.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:27:39.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:27:49.757 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:27:49.789 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:27:49.789 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:27:49.789 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:27:49.789 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:27:49.789 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:27:49.789 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:27:49.789 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:27:49.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:27:49.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:27:49.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:28:09.792 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:28:09.826 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:28:09.826 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:28:09.826 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:28:09.826 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:28:09.826 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:28:09.826 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:28:09.826 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:28:09.828 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:28:09.828 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:28:09.829 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:28:09.829 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:28:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:28:09.829 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:28:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:28:09.829 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:28:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:28:09.829 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:28:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:28:09.829 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:28:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:28:09.829 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:28:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:28:09.829 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:28:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:28:09.829 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:28:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:28:09.829 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:28:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:28:09.829 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:28:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:28:09.829 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:28:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:28:09.829 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:28:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:28:09.829 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:28:09.829 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:28:09.829 139328 ERROR oslo_service.periodic_task >2018-06-25 04:28:39.710 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:28:39.711 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:28:39.718 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:28:39.719 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:28:39.751 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:28:39.751 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:28:39.751 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:28:39.751 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:28:39.751 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:28:39.751 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:28:39.751 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:28:39.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:28:39.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:28:39.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:28:49.754 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:28:49.789 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:28:49.789 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:28:49.789 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:28:49.789 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:28:49.789 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:28:49.789 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:28:49.789 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:28:49.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:28:49.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:28:49.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:29:09.792 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:29:09.826 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:29:09.826 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:29:09.826 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:29:09.826 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:29:09.826 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:29:09.826 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:29:09.826 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:29:09.829 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:29:09.830 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:29:09.830 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:29:09.830 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:29:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:29:09.830 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:29:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:29:09.830 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:29:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:29:09.830 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:29:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:29:09.830 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:29:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:29:09.830 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:29:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:29:09.830 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:29:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:29:09.830 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:29:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:29:09.830 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:29:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:29:09.830 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:29:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:29:09.830 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:29:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:29:09.830 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:29:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:29:09.830 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:29:09.830 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:29:09.830 139328 ERROR oslo_service.periodic_task >2018-06-25 04:29:39.712 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:29:39.713 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:29:39.719 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:29:39.720 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:29:39.752 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:29:39.752 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:29:39.752 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:29:39.752 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:29:39.752 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:29:39.752 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:29:39.752 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:29:39.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:29:39.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:29:39.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:29:49.756 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:29:49.790 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:29:49.790 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:29:49.790 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:29:49.790 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:29:49.790 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:29:49.790 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:29:49.790 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:29:49.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:29:49.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:29:49.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:30:09.794 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:30:09.829 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:30:09.829 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:30:09.829 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:30:09.829 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:30:09.829 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:30:09.829 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:30:09.829 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:30:09.831 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:30:09.831 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:30:09.832 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:30:09.832 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:30:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:30:09.832 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:30:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:30:09.832 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:30:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:30:09.832 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:30:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:30:09.832 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:30:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:30:09.832 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:30:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:30:09.832 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:30:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:30:09.832 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:30:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:30:09.832 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:30:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:30:09.832 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:30:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:30:09.832 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:30:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:30:09.832 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:30:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:30:09.832 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:30:09.832 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:30:09.832 139328 ERROR oslo_service.periodic_task >2018-06-25 04:30:39.713 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:30:39.714 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:30:39.727 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:30:39.728 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:30:39.761 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:30:39.761 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:30:39.761 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:30:39.761 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:30:39.761 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:30:39.761 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:30:39.761 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:30:39.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:30:39.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:30:39.764 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:30:49.764 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:30:49.806 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:30:49.806 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:30:49.806 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:30:49.806 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:30:49.806 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:30:49.806 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:30:49.806 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:30:49.808 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:30:49.808 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.08 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:30:49.808 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:31:09.809 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:31:09.844 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:31:09.844 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:31:09.844 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:31:09.844 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:31:09.844 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:31:09.844 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:31:09.844 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:31:09.846 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:31:09.846 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.118 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:31:09.846 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:31:09.846 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:31:09.846 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:31:09.846 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:31:09.846 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:31:09.846 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:31:09.846 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:31:09.846 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:31:09.846 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:31:09.846 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:31:09.846 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:31:09.846 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:31:09.846 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:31:09.846 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:31:09.846 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:31:09.846 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:31:09.846 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:31:09.846 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:31:09.846 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:31:09.846 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:31:09.846 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:31:09.846 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:31:09.846 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:31:09.846 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:31:09.846 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:31:09.846 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:31:09.846 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:31:09.846 139328 ERROR oslo_service.periodic_task >2018-06-25 04:31:39.718 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:31:39.718 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:31:39.725 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:31:39.726 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:31:39.760 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:31:39.760 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:31:39.760 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:31:39.760 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:31:39.760 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:31:39.760 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:31:39.760 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:31:39.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:31:39.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:31:39.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:31:49.763 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:31:49.797 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:31:49.797 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:31:49.797 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:31:49.797 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:31:49.797 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:31:49.797 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:31:49.797 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:31:49.799 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:31:49.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:31:49.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:32:09.801 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:32:09.839 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:32:09.839 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:32:09.839 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:32:09.839 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:32:09.839 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:32:09.839 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:32:09.839 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:32:09.841 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:32:09.842 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:32:09.842 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:32:09.842 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:32:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:32:09.842 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:32:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:32:09.842 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:32:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:32:09.842 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:32:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:32:09.842 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:32:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:32:09.842 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:32:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:32:09.842 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:32:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:32:09.842 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:32:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:32:09.842 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:32:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:32:09.842 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:32:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:32:09.842 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:32:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:32:09.842 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:32:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:32:09.842 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:32:09.842 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:32:09.842 139328 ERROR oslo_service.periodic_task >2018-06-25 04:32:39.713 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:32:39.714 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:32:39.720 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:32:39.721 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:32:39.752 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:32:39.752 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:32:39.752 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:32:39.752 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:32:39.752 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:32:39.752 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:32:39.752 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:32:39.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:32:39.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:32:39.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:32:49.755 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:32:49.789 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:32:49.789 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:32:49.789 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:32:49.789 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:32:49.789 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:32:49.789 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:32:49.789 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:32:49.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:32:49.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:32:49.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:33:09.792 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:33:09.828 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:33:09.828 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:33:09.828 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:33:09.828 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:33:09.828 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:33:09.828 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:33:09.828 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:33:09.831 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:33:09.831 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:33:09.831 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:33:09.831 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:33:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:33:09.831 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:33:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:33:09.831 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:33:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:33:09.831 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:33:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:33:09.831 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:33:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:33:09.831 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:33:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:33:09.831 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:33:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:33:09.831 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:33:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:33:09.831 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:33:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:33:09.831 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:33:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:33:09.831 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:33:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:33:09.831 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:33:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:33:09.831 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:33:09.831 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:33:09.831 139328 ERROR oslo_service.periodic_task >2018-06-25 04:33:39.713 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:33:39.713 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:33:39.720 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:33:39.720 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:33:39.753 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:33:39.753 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:33:39.753 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:33:39.753 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:33:39.753 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:33:39.753 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:33:39.753 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:33:39.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:33:39.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:33:39.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:33:49.756 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:33:49.791 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:33:49.791 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:33:49.791 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:33:49.791 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:33:49.791 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:33:49.791 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:33:49.791 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:33:49.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:33:49.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:33:49.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:34:09.795 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:34:09.828 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:34:09.828 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:34:09.828 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:34:09.828 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:34:09.828 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:34:09.828 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:34:09.828 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:34:09.830 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:34:09.831 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:34:09.831 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:34:09.831 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:34:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:34:09.831 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:34:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:34:09.831 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:34:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:34:09.831 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:34:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:34:09.831 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:34:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:34:09.831 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:34:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:34:09.831 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:34:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:34:09.831 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:34:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:34:09.831 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:34:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:34:09.831 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:34:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:34:09.831 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:34:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:34:09.831 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:34:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:34:09.831 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:34:09.831 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:34:09.831 139328 ERROR oslo_service.periodic_task >2018-06-25 04:34:39.713 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:34:39.713 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:34:39.720 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:34:39.721 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:34:39.754 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:34:39.754 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:34:39.754 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:34:39.754 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:34:39.754 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:34:39.754 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:34:39.754 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:34:39.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:34:39.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:34:39.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:34:49.757 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:34:49.792 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:34:49.792 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:34:49.792 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:34:49.792 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:34:49.792 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:34:49.792 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:34:49.792 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:34:49.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:34:49.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:34:49.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:35:09.796 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:35:09.832 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:35:09.832 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:35:09.832 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:35:09.832 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:35:09.832 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:35:09.832 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:35:09.832 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:35:09.835 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:35:09.836 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:35:09.836 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:35:09.836 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:35:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:35:09.836 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:35:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:35:09.836 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:35:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:35:09.836 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:35:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:35:09.836 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:35:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:35:09.836 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:35:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:35:09.836 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:35:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:35:09.836 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:35:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:35:09.836 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:35:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:35:09.836 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:35:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:35:09.836 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:35:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:35:09.836 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:35:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:35:09.836 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:35:09.836 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:35:09.836 139328 ERROR oslo_service.periodic_task >2018-06-25 04:35:39.718 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:35:39.719 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:35:39.730 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:35:39.731 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:35:39.765 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:35:39.765 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:35:39.765 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:35:39.765 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:35:39.765 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:35:39.765 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:35:39.765 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:35:39.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:35:39.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:35:39.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:35:49.768 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:35:49.804 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:35:49.804 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:35:49.804 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:35:49.804 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:35:49.804 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:35:49.804 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:35:49.804 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:35:49.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:35:49.807 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:35:49.807 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:36:09.808 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:36:09.842 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:36:09.842 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:36:09.842 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:36:09.842 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:36:09.842 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:36:09.842 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:36:09.842 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:36:09.844 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:36:09.844 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:36:09.845 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:36:09.845 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:36:09.845 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:36:09.845 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:36:09.845 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:36:09.845 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:36:09.845 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:36:09.845 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:36:09.845 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:36:09.845 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:36:09.845 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:36:09.845 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:36:09.845 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:36:09.845 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:36:09.845 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:36:09.845 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:36:09.845 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:36:09.845 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:36:09.845 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:36:09.845 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:36:09.845 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:36:09.845 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:36:09.845 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:36:09.845 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:36:09.845 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:36:09.845 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:36:09.845 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:36:09.845 139328 ERROR oslo_service.periodic_task >2018-06-25 04:36:39.716 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:36:39.717 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:36:39.724 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:36:39.725 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:36:39.758 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:36:39.758 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:36:39.758 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:36:39.758 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:36:39.758 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:36:39.758 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:36:39.758 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:36:39.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:36:39.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:36:39.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:36:49.761 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:36:49.797 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:36:49.797 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:36:49.797 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:36:49.797 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:36:49.797 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:36:49.797 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:36:49.797 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:36:49.801 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:36:49.801 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:36:49.801 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:37:09.802 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:37:09.837 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:37:09.837 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:37:09.837 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:37:09.837 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:37:09.837 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:37:09.837 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:37:09.837 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:37:09.840 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:37:09.840 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:37:09.841 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:37:09.841 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:37:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:37:09.841 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:37:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:37:09.841 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:37:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:37:09.841 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:37:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:37:09.841 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:37:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:37:09.841 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:37:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:37:09.841 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:37:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:37:09.841 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:37:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:37:09.841 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:37:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:37:09.841 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:37:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:37:09.841 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:37:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:37:09.841 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:37:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:37:09.841 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:37:09.841 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:37:09.841 139328 ERROR oslo_service.periodic_task >2018-06-25 04:37:39.712 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:37:39.713 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:37:39.721 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:37:39.722 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:37:39.756 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:37:39.756 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:37:39.756 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:37:39.756 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:37:39.756 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:37:39.756 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:37:39.756 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:37:39.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:37:39.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:37:39.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:37:49.760 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:37:49.798 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:37:49.798 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:37:49.798 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:37:49.798 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:37:49.798 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:37:49.798 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:37:49.798 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:37:49.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:37:49.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:37:49.801 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:38:09.801 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:38:09.837 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:38:09.837 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:38:09.837 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:38:09.837 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:38:09.837 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:38:09.837 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:38:09.837 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:38:09.840 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:38:09.841 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.119 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:38:09.841 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:38:09.841 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:38:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:38:09.841 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:38:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:38:09.841 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:38:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:38:09.841 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:38:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:38:09.841 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:38:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:38:09.841 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:38:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:38:09.841 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:38:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:38:09.841 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:38:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:38:09.841 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:38:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:38:09.841 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:38:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:38:09.841 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:38:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:38:09.841 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:38:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:38:09.841 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:38:09.841 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:38:09.841 139328 ERROR oslo_service.periodic_task >2018-06-25 04:38:39.712 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:38:39.713 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:38:39.720 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:38:39.720 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:38:39.754 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:38:39.754 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:38:39.754 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:38:39.754 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:38:39.754 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:38:39.754 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:38:39.754 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:38:39.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:38:39.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:38:39.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:38:49.757 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:38:49.791 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:38:49.791 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:38:49.791 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:38:49.791 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:38:49.791 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:38:49.791 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:38:49.791 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:38:49.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:38:49.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:38:49.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:39:09.795 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:39:09.828 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:39:09.828 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:39:09.828 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:39:09.828 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:39:09.828 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:39:09.828 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:39:09.828 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:39:09.829 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:39:09.830 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:39:09.830 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:39:09.830 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:39:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:39:09.830 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:39:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:39:09.830 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:39:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:39:09.830 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:39:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:39:09.830 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:39:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:39:09.830 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:39:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:39:09.830 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:39:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:39:09.830 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:39:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:39:09.830 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:39:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:39:09.830 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:39:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:39:09.830 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:39:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:39:09.830 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:39:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:39:09.830 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:39:09.830 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:39:09.830 139328 ERROR oslo_service.periodic_task >2018-06-25 04:39:39.711 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:39:39.712 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:39:39.718 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:39:39.719 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:39:39.752 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:39:39.752 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:39:39.752 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:39:39.752 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:39:39.752 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:39:39.752 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:39:39.752 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:39:39.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:39:39.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:39:39.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:39:49.757 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:39:49.792 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:39:49.792 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:39:49.792 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:39:49.792 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:39:49.792 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:39:49.792 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:39:49.792 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:39:49.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:39:49.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:39:49.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:40:09.795 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:40:09.828 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:40:09.828 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:40:09.828 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:40:09.828 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:40:09.828 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:40:09.828 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:40:09.828 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:40:09.830 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:40:09.831 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:40:09.831 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:40:09.831 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:40:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:40:09.831 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:40:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:40:09.831 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:40:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:40:09.831 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:40:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:40:09.831 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:40:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:40:09.831 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:40:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:40:09.831 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:40:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:40:09.831 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:40:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:40:09.831 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:40:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:40:09.831 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:40:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:40:09.831 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:40:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:40:09.831 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:40:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:40:09.831 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:40:09.831 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:40:09.831 139328 ERROR oslo_service.periodic_task >2018-06-25 04:40:39.713 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:40:39.714 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:40:39.720 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:40:39.721 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:40:39.760 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:40:39.760 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:40:39.760 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:40:39.760 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:40:39.760 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:40:39.760 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:40:39.760 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:40:39.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:40:39.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.042 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:40:39.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:40:49.764 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:40:49.797 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:40:49.797 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:40:49.797 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:40:49.797 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:40:49.797 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:40:49.797 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:40:49.797 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:40:49.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:40:49.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:40:49.801 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:41:09.801 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:41:09.839 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:41:09.839 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:41:09.839 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:41:09.839 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:41:09.839 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:41:09.839 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:41:09.839 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:41:09.841 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:41:09.841 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.12 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:41:09.841 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:41:09.841 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:41:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:41:09.841 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:41:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:41:09.841 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:41:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:41:09.841 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:41:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:41:09.841 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:41:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:41:09.841 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:41:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:41:09.841 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:41:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:41:09.841 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:41:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:41:09.841 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:41:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:41:09.841 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:41:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:41:09.841 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:41:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:41:09.841 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:41:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:41:09.841 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:41:09.841 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:41:09.841 139328 ERROR oslo_service.periodic_task >2018-06-25 04:41:39.713 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:41:39.713 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:42:39.711 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:42:39.712 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:42:39.719 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:42:39.720 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:42:39.754 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:42:39.754 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:42:39.754 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:42:39.754 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:42:39.754 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:42:39.754 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:42:39.754 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:42:39.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:42:39.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:42:39.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:42:49.758 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:42:49.791 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:42:49.791 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:42:49.791 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:42:49.791 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:42:49.791 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:42:49.791 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:42:49.791 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:42:49.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:42:49.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:42:49.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:43:09.794 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:43:09.828 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:43:09.828 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:43:09.828 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:43:09.828 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:43:09.828 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:43:09.828 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:43:09.828 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:43:09.830 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:43:09.830 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:43:09.831 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:43:09.831 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:43:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:43:09.831 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:43:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:43:09.831 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:43:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:43:09.831 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:43:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:43:09.831 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:43:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:43:09.831 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:43:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:43:09.831 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:43:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:43:09.831 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:43:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:43:09.831 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:43:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:43:09.831 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:43:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:43:09.831 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:43:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:43:09.831 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:43:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:43:09.831 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:43:09.831 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:43:09.831 139328 ERROR oslo_service.periodic_task >2018-06-25 04:43:39.713 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:43:39.714 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:43:39.721 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:43:39.721 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:43:39.755 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:43:39.755 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:43:39.755 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:43:39.755 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:43:39.755 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:43:39.755 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:43:39.755 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:43:39.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:43:39.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:43:39.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:43:49.758 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:43:49.794 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:43:49.794 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:43:49.794 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:43:49.794 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:43:49.794 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:43:49.794 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:43:49.794 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:43:49.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:43:49.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:43:49.797 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:44:09.797 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:44:09.830 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:44:09.830 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:44:09.830 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:44:09.830 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:44:09.830 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:44:09.830 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:44:09.830 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:44:09.833 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:44:09.833 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:44:09.834 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:44:09.834 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:44:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:44:09.834 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:44:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:44:09.834 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:44:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:44:09.834 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:44:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:44:09.834 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:44:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:44:09.834 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:44:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:44:09.834 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:44:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:44:09.834 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:44:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:44:09.834 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:44:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:44:09.834 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:44:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:44:09.834 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:44:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:44:09.834 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:44:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:44:09.834 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:44:09.834 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:44:09.834 139328 ERROR oslo_service.periodic_task >2018-06-25 04:44:39.715 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:44:39.716 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:44:39.722 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:44:39.723 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:44:39.756 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:44:39.756 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:44:39.756 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:44:39.756 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:44:39.756 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:44:39.756 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:44:39.756 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:44:39.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:44:39.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:44:39.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:44:49.759 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:44:49.791 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:44:49.791 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:44:49.791 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:44:49.791 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:44:49.791 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:44:49.791 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:44:49.791 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:44:49.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:44:49.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:44:49.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:45:09.795 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:45:09.828 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:45:09.828 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:45:09.828 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:45:09.828 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:45:09.828 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:45:09.828 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:45:09.828 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:45:09.831 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:45:09.831 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:45:09.832 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:45:09.832 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:45:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:45:09.832 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:45:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:45:09.832 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:45:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:45:09.832 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:45:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:45:09.832 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:45:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:45:09.832 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:45:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:45:09.832 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:45:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:45:09.832 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:45:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:45:09.832 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:45:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:45:09.832 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:45:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:45:09.832 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:45:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:45:09.832 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:45:09.832 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:45:09.832 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:45:09.832 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:45:09.832 139328 ERROR oslo_service.periodic_task >2018-06-25 04:45:39.713 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:45:39.714 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:45:39.724 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:45:39.724 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:45:39.760 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:45:39.760 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:45:39.760 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:45:39.760 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:45:39.760 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:45:39.760 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:45:39.760 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:45:39.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:45:39.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:45:39.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:45:49.763 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:45:49.797 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:45:49.797 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:45:49.797 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:45:49.797 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:45:49.797 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:45:49.797 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:45:49.797 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:45:49.799 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:45:49.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:45:49.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:46:09.801 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:46:09.848 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:46:09.848 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:46:09.848 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:46:09.848 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:46:09.848 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:46:09.848 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:46:09.848 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:46:09.850 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:46:09.850 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.125 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:46:09.851 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:46:09.851 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:46:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:46:09.851 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:46:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:46:09.851 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:46:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:46:09.851 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:46:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:46:09.851 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:46:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:46:09.851 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:46:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:46:09.851 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:46:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:46:09.851 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:46:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:46:09.851 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:46:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:46:09.851 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:46:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:46:09.851 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:46:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:46:09.851 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:46:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:46:09.851 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:46:09.851 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:46:09.851 139328 ERROR oslo_service.periodic_task >2018-06-25 04:46:39.712 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:46:39.713 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:46:39.719 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:46:39.719 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:46:39.754 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:46:39.754 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:46:39.754 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:46:39.754 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:46:39.754 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:46:39.754 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:46:39.754 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:46:39.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:46:39.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:46:39.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:46:49.757 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:46:49.793 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:46:49.793 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:46:49.793 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:46:49.793 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:46:49.793 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:46:49.793 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:46:49.793 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:46:49.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:46:49.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:46:49.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:47:09.797 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:47:09.829 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:47:09.829 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:47:09.829 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:47:09.829 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:47:09.829 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:47:09.829 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:47:09.829 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:47:09.831 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:47:09.831 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:47:09.831 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:47:09.831 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:47:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:47:09.831 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:47:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:47:09.831 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:47:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:47:09.831 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:47:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:47:09.831 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:47:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:47:09.831 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:47:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:47:09.831 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:47:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:47:09.831 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:47:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:47:09.831 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:47:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:47:09.831 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:47:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:47:09.831 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:47:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:47:09.831 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:47:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:47:09.831 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:47:09.831 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:47:09.831 139328 ERROR oslo_service.periodic_task >2018-06-25 04:47:39.713 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:47:39.713 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:47:39.721 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:47:39.722 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:47:39.759 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:47:39.759 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:47:39.759 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:47:39.759 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:47:39.759 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:47:39.759 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:47:39.759 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:47:39.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:47:39.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:47:39.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:47:49.762 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:47:49.794 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:47:49.794 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:47:49.794 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:47:49.794 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:47:49.794 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:47:49.794 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:47:49.794 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:47:49.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:47:49.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:47:49.797 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:48:09.797 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:48:09.834 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:48:09.834 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:48:09.834 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:48:09.834 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:48:09.834 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:48:09.834 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:48:09.834 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:48:09.836 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:48:09.837 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:48:09.837 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:48:09.837 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:48:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:48:09.837 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:48:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:48:09.837 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:48:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:48:09.837 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:48:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:48:09.837 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:48:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:48:09.837 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:48:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:48:09.837 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:48:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:48:09.837 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:48:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:48:09.837 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:48:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:48:09.837 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:48:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:48:09.837 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:48:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:48:09.837 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:48:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:48:09.837 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:48:09.837 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:48:09.837 139328 ERROR oslo_service.periodic_task >2018-06-25 04:48:39.709 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:48:39.710 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:48:39.716 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:48:39.716 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:48:39.750 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:48:39.750 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:48:39.750 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:48:39.750 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:48:39.750 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:48:39.750 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:48:39.750 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:48:39.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:48:39.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:48:39.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:48:49.753 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:48:49.787 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:48:49.787 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:48:49.787 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:48:49.787 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:48:49.787 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:48:49.787 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:48:49.787 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:48:49.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:48:49.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:48:49.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:49:09.790 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:49:09.822 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:49:09.822 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:49:09.822 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:49:09.822 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:49:09.822 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:49:09.822 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:49:09.822 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:49:09.824 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:49:09.824 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.107 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:49:09.825 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:49:09.825 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:49:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:49:09.825 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:49:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:49:09.825 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:49:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:49:09.825 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:49:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:49:09.825 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:49:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:49:09.825 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:49:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:49:09.825 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:49:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:49:09.825 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:49:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:49:09.825 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:49:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:49:09.825 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:49:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:49:09.825 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:49:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:49:09.825 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:49:09.825 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:49:09.825 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:49:09.825 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:49:09.825 139328 ERROR oslo_service.periodic_task >2018-06-25 04:49:39.706 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:49:39.706 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:49:39.713 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:49:39.714 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:49:39.748 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:49:39.748 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:49:39.748 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:49:39.748 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:49:39.748 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:49:39.748 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:49:39.748 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:49:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:49:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:49:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:49:49.751 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:49:49.783 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:49:49.783 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:49:49.783 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:49:49.783 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:49:49.783 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:49:49.783 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:49:49.783 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:49:49.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:49:49.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:49:49.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:50:09.786 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:50:09.823 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:50:09.823 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:50:09.823 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:50:09.823 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:50:09.823 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:50:09.823 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:50:09.823 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:50:09.826 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:50:09.826 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:50:09.826 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:50:09.826 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:50:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:50:09.826 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:50:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:50:09.826 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:50:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:50:09.826 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:50:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:50:09.826 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:50:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:50:09.826 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:50:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:50:09.826 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:50:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:50:09.826 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:50:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:50:09.826 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:50:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:50:09.826 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:50:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:50:09.826 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:50:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:50:09.826 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:50:09.826 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:50:09.826 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:50:09.826 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:50:09.826 139328 ERROR oslo_service.periodic_task >2018-06-25 04:50:39.708 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:50:39.708 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:50:39.715 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:50:39.715 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:50:39.750 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:50:39.750 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:50:39.750 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:50:39.750 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:50:39.750 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:50:39.750 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:50:39.750 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:50:39.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:50:39.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:50:39.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:50:49.754 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:50:49.788 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:50:49.788 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:50:49.788 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:50:49.788 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:50:49.788 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:50:49.788 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:50:49.788 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:50:49.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:50:49.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:50:49.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:51:09.792 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:51:09.827 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:51:09.827 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:51:09.827 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:51:09.827 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:51:09.827 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:51:09.827 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:51:09.827 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:51:09.829 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:51:09.829 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:51:09.830 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:51:09.830 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:51:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:51:09.830 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:51:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:51:09.830 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:51:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:51:09.830 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:51:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:51:09.830 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:51:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:51:09.830 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:51:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:51:09.830 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:51:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:51:09.830 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:51:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:51:09.830 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:51:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:51:09.830 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:51:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:51:09.830 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:51:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:51:09.830 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:51:09.830 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:51:09.830 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:51:09.830 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:51:09.830 139328 ERROR oslo_service.periodic_task >2018-06-25 04:51:39.711 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:51:39.712 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:51:39.718 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:51:39.719 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:51:39.750 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:51:39.750 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:51:39.750 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:51:39.750 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:51:39.750 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:51:39.750 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:51:39.750 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:51:39.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:51:39.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:51:39.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:51:49.753 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:51:49.789 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:51:49.789 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:51:49.789 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:51:49.789 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:51:49.789 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:51:49.789 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:51:49.789 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:51:49.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:51:49.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:51:49.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:52:09.792 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:52:09.825 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:52:09.825 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:52:09.825 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:52:09.825 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:52:09.825 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:52:09.825 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:52:09.825 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:52:09.827 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:52:09.828 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:52:09.828 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:52:09.828 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:52:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:52:09.828 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:52:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:52:09.828 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:52:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:52:09.828 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:52:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:52:09.828 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:52:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:52:09.828 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:52:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:52:09.828 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:52:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:52:09.828 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:52:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:52:09.828 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:52:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:52:09.828 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:52:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:52:09.828 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:52:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:52:09.828 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:52:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:52:09.828 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:52:09.828 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:52:09.828 139328 ERROR oslo_service.periodic_task >2018-06-25 04:52:39.709 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:52:39.710 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:52:39.718 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:52:39.718 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:52:39.752 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:52:39.752 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:52:39.752 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:52:39.752 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:52:39.752 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:52:39.752 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:52:39.752 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:52:39.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:52:39.754 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:52:39.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:52:49.755 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:52:49.791 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:52:49.791 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:52:49.791 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:52:49.791 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:52:49.791 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:52:49.791 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:52:49.791 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:52:49.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:52:49.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:52:49.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:53:09.794 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:53:09.826 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:53:09.826 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:53:09.826 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:53:09.826 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:53:09.826 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:53:09.826 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:53:09.826 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:53:09.828 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:53:09.828 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:53:09.829 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:53:09.829 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:53:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:53:09.829 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:53:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:53:09.829 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:53:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:53:09.829 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:53:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:53:09.829 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:53:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:53:09.829 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:53:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:53:09.829 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:53:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:53:09.829 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:53:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:53:09.829 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:53:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:53:09.829 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:53:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:53:09.829 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:53:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:53:09.829 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:53:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:53:09.829 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:53:09.829 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:53:09.829 139328 ERROR oslo_service.periodic_task >2018-06-25 04:53:39.710 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:53:39.711 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:53:39.720 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:53:39.721 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:53:39.753 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:53:39.753 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:53:39.753 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:53:39.753 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:53:39.753 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:53:39.753 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:53:39.753 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:53:39.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:53:39.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:53:39.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:53:49.757 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:53:49.791 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:53:49.791 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:53:49.791 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:53:49.791 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:53:49.791 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:53:49.791 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:53:49.791 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:53:49.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:53:49.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:53:49.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:54:09.794 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:54:09.827 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:54:09.827 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:54:09.827 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:54:09.827 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:54:09.827 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:54:09.827 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:54:09.827 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:54:09.829 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:54:09.829 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:54:09.829 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:54:09.829 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:54:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:54:09.829 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:54:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:54:09.829 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:54:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:54:09.829 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:54:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:54:09.829 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:54:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:54:09.829 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:54:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:54:09.829 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:54:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:54:09.829 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:54:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:54:09.829 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:54:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:54:09.829 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:54:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:54:09.829 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:54:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:54:09.829 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:54:09.829 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:54:09.829 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:54:09.829 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:54:09.829 139328 ERROR oslo_service.periodic_task >2018-06-25 04:54:39.711 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:54:39.711 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:54:39.720 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:54:39.721 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:54:39.762 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:54:39.762 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:54:39.762 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:54:39.762 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:54:39.762 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:54:39.762 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:54:39.762 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:54:39.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:54:39.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.044 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:54:39.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:54:49.767 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:54:49.801 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:54:49.801 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:54:49.801 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:54:49.801 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:54:49.801 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:54:49.801 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:54:49.801 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:54:49.803 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:54:49.804 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.083 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:54:49.804 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:55:09.805 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:55:09.839 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:55:09.839 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:55:09.839 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:55:09.839 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:55:09.839 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:55:09.839 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:55:09.839 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:55:09.841 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:55:09.841 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.12 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:55:09.842 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:55:09.842 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:55:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:55:09.842 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:55:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:55:09.842 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:55:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:55:09.842 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:55:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:55:09.842 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:55:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:55:09.842 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:55:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:55:09.842 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:55:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:55:09.842 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:55:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:55:09.842 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:55:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:55:09.842 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:55:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:55:09.842 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:55:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:55:09.842 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:55:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:55:09.842 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:55:09.842 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:55:09.842 139328 ERROR oslo_service.periodic_task >2018-06-25 04:55:39.713 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:55:39.714 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:55:39.724 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:55:39.724 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:55:39.758 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:55:39.758 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:55:39.758 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:55:39.758 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:55:39.758 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:55:39.758 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:55:39.758 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:55:39.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:55:39.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:55:39.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:55:49.761 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:55:49.814 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:55:49.814 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:55:49.814 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:55:49.814 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:55:49.814 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:55:49.814 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:55:49.814 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:55:49.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:55:49.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.092 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:55:49.818 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:56:09.818 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:56:09.851 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:56:09.851 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:56:09.851 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:56:09.851 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:56:09.851 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:56:09.851 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:56:09.851 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:56:09.854 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:56:09.854 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.129 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:56:09.854 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:56:09.854 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:56:09.854 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:56:09.854 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:56:09.854 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:56:09.854 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:56:09.854 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:56:09.854 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:56:09.854 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:56:09.854 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:56:09.854 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:56:09.854 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:56:09.854 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:56:09.854 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:56:09.854 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:56:09.854 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:56:09.854 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:56:09.854 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:56:09.854 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:56:09.854 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:56:09.854 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:56:09.854 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:56:09.854 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:56:09.854 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:56:09.854 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:56:09.854 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:56:09.854 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:56:09.854 139328 ERROR oslo_service.periodic_task >2018-06-25 04:56:39.716 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:56:39.717 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:56:39.724 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:56:39.724 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:56:39.757 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:56:39.757 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:56:39.757 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:56:39.757 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:56:39.757 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:56:39.757 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:56:39.757 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:56:39.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:56:39.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:56:39.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:56:49.761 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:56:49.797 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:56:49.797 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:56:49.797 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:56:49.797 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:56:49.797 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:56:49.797 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:56:49.797 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:56:49.799 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:56:49.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:56:49.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:57:09.801 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:57:09.849 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:57:09.849 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:57:09.849 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:57:09.849 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:57:09.849 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:57:09.849 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:57:09.849 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:57:09.851 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:57:09.851 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.126 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:57:09.852 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:57:09.852 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:57:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:57:09.852 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:57:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:57:09.852 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:57:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:57:09.852 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:57:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:57:09.852 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:57:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:57:09.852 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:57:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:57:09.852 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:57:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:57:09.852 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:57:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:57:09.852 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:57:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:57:09.852 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:57:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:57:09.852 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:57:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:57:09.852 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:57:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:57:09.852 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:57:09.852 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:57:09.852 139328 ERROR oslo_service.periodic_task >2018-06-25 04:57:39.713 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:57:39.714 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:57:39.721 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:57:39.722 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:57:39.758 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:57:39.758 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:57:39.758 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:57:39.758 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:57:39.758 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:57:39.758 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:57:39.758 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:57:39.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:57:39.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:57:39.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:57:49.761 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:57:49.815 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:57:49.815 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:57:49.815 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:57:49.815 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:57:49.815 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:57:49.815 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:57:49.815 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:57:49.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:57:49.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.096 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:57:49.818 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:58:09.818 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:58:09.852 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:58:09.852 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:58:09.852 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:58:09.852 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:58:09.852 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:58:09.852 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:58:09.852 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:58:09.854 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:58:09.855 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.133 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:58:09.855 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:58:09.855 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:58:09.855 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:58:09.855 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:58:09.855 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:58:09.855 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:58:09.855 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:58:09.855 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:58:09.855 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:58:09.855 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:58:09.855 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:58:09.855 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:58:09.855 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:58:09.855 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:58:09.855 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:58:09.855 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:58:09.855 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:58:09.855 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:58:09.855 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:58:09.855 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:58:09.855 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:58:09.855 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:58:09.855 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:58:09.855 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:58:09.855 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:58:09.855 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:58:09.855 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:58:09.855 139328 ERROR oslo_service.periodic_task >2018-06-25 04:58:39.716 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:58:39.717 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:58:39.723 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:58:39.724 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:58:39.756 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:58:39.756 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:58:39.756 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:58:39.756 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:58:39.756 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:58:39.756 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:58:39.756 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:58:39.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:58:39.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:58:39.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:58:49.759 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:58:49.792 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:58:49.792 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:58:49.792 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:58:49.792 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:58:49.792 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:58:49.792 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:58:49.792 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:58:49.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:58:49.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:58:49.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:59:09.796 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:59:09.830 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:59:09.830 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:59:09.830 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:59:09.830 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:59:09.830 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:59:09.830 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:59:09.830 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:59:09.832 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:59:09.833 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:59:09.833 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:59:09.833 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 04:59:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 04:59:09.833 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 04:59:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 04:59:09.833 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 04:59:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 04:59:09.833 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 04:59:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 04:59:09.833 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 04:59:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 04:59:09.833 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 04:59:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 04:59:09.833 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 04:59:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 04:59:09.833 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 04:59:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 04:59:09.833 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 04:59:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 04:59:09.833 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 04:59:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 04:59:09.833 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 04:59:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 04:59:09.833 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 04:59:09.833 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 04:59:09.833 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 04:59:09.833 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 04:59:09.833 139328 ERROR oslo_service.periodic_task >2018-06-25 04:59:39.715 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:59:39.715 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 04:59:39.721 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 04:59:39.722 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:59:39.755 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:59:39.755 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:59:39.755 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:59:39.755 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:59:39.755 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:59:39.755 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:59:39.755 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:59:39.757 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:59:39.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:59:39.758 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 04:59:49.758 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 04:59:49.797 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 04:59:49.797 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 04:59:49.797 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 04:59:49.797 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 04:59:49.797 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 04:59:49.797 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 04:59:49.797 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 04:59:49.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 04:59:49.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 04:59:49.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:00:09.801 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:00:09.836 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:00:09.836 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:00:09.836 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:00:09.836 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:00:09.836 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:00:09.836 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:00:09.836 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:00:09.839 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:00:09.839 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:00:09.839 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:00:09.839 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:00:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:00:09.839 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:00:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:00:09.839 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:00:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:00:09.839 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:00:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:00:09.839 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:00:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:00:09.839 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:00:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:00:09.839 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:00:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:00:09.839 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:00:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:00:09.839 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:00:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:00:09.839 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:00:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:00:09.839 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:00:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:00:09.839 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:00:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:00:09.839 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:00:09.839 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:00:09.839 139328 ERROR oslo_service.periodic_task >2018-06-25 05:00:39.711 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:00:39.712 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:00:39.747 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:00:39.747 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:00:39.747 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:00:39.747 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:00:39.747 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:00:39.747 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:00:39.747 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:00:39.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:00:39.749 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:00:39.750 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:00:49.750 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:00:49.786 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:00:49.786 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:00:49.786 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:00:49.786 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:00:49.786 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:00:49.786 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:00:49.786 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:00:49.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:00:49.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:00:49.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:01:09.789 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:01:09.825 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:01:09.825 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:01:09.825 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:01:09.825 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:01:09.825 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:01:09.825 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:01:09.825 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:01:09.827 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:01:09.828 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:01:09.828 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:01:09.828 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:01:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:01:09.828 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:01:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:01:09.828 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:01:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:01:09.828 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:01:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:01:09.828 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:01:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:01:09.828 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:01:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:01:09.828 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:01:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:01:09.828 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:01:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:01:09.828 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:01:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:01:09.828 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:01:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:01:09.828 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:01:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:01:09.828 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:01:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:01:09.828 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:01:09.828 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:01:09.828 139328 ERROR oslo_service.periodic_task >2018-06-25 05:01:39.709 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:01:39.710 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:01:39.718 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:01:39.719 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:01:39.752 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:01:39.752 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:01:39.752 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:01:39.752 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:01:39.752 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:01:39.752 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:01:39.752 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:01:39.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:01:39.755 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:01:39.756 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:01:49.756 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:01:49.796 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:01:49.796 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:01:49.796 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:01:49.796 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:01:49.796 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:01:49.796 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:01:49.796 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:01:49.799 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:01:49.799 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.08 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:01:49.799 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:02:09.800 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:02:09.836 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:02:09.836 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:02:09.836 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:02:09.836 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:02:09.836 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:02:09.836 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:02:09.836 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:02:09.838 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:02:09.838 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.119 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:02:09.839 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:02:09.839 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:02:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:02:09.839 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:02:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:02:09.839 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:02:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:02:09.839 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:02:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:02:09.839 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:02:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:02:09.839 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:02:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:02:09.839 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:02:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:02:09.839 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:02:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:02:09.839 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:02:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:02:09.839 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:02:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:02:09.839 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:02:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:02:09.839 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:02:09.839 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:02:09.839 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:02:09.839 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:02:09.839 139328 ERROR oslo_service.periodic_task >2018-06-25 05:02:39.710 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:02:39.711 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:02:39.718 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:02:39.719 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:02:39.760 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:02:39.760 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:02:39.760 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:02:39.760 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:02:39.760 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:02:39.760 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:02:39.760 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:02:39.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:02:39.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.044 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:02:39.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:02:49.764 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:02:49.800 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:02:49.800 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:02:49.800 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:02:49.800 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:02:49.800 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:02:49.800 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:02:49.800 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:02:49.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:02:49.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.083 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:02:49.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:03:09.803 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:03:09.836 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:03:09.836 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:03:09.836 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:03:09.836 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:03:09.836 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:03:09.836 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:03:09.836 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:03:09.837 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:03:09.838 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.119 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:03:09.838 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:03:09.838 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:03:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:03:09.838 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:03:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:03:09.838 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:03:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:03:09.838 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:03:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:03:09.838 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:03:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:03:09.838 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:03:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:03:09.838 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:03:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:03:09.838 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:03:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:03:09.838 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:03:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:03:09.838 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:03:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:03:09.838 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:03:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:03:09.838 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:03:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:03:09.838 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:03:09.838 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:03:09.838 139328 ERROR oslo_service.periodic_task >2018-06-25 05:03:39.710 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:03:39.710 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:03:39.723 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:03:39.724 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:03:39.760 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:03:39.760 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:03:39.760 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:03:39.760 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:03:39.760 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:03:39.760 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:03:39.760 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:03:39.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:03:39.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:03:39.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:03:49.764 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:03:49.799 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:03:49.799 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:03:49.799 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:03:49.799 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:03:49.799 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:03:49.799 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:03:49.799 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:03:49.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:03:49.801 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:03:49.801 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:04:09.802 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:04:09.835 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:04:09.835 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:04:09.835 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:04:09.835 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:04:09.835 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:04:09.835 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:04:09.835 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:04:09.837 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:04:09.837 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:04:09.838 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:04:09.838 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:04:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:04:09.838 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:04:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:04:09.838 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:04:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:04:09.838 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:04:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:04:09.838 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:04:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:04:09.838 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:04:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:04:09.838 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:04:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:04:09.838 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:04:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:04:09.838 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:04:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:04:09.838 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:04:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:04:09.838 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:04:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:04:09.838 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:04:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:04:09.838 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:04:09.838 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:04:09.838 139328 ERROR oslo_service.periodic_task >2018-06-25 05:04:39.709 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:04:39.709 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:04:39.716 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:04:39.716 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:04:39.749 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:04:39.749 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:04:39.749 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:04:39.749 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:04:39.749 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:04:39.749 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:04:39.749 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:04:39.751 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:04:39.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:04:39.752 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:04:49.753 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:04:49.788 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:04:49.788 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:04:49.788 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:04:49.788 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:04:49.788 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:04:49.788 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:04:49.788 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:04:49.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:04:49.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:04:49.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:05:09.792 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:05:09.825 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:05:09.825 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:05:09.825 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:05:09.825 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:05:09.825 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:05:09.825 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:05:09.825 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:05:09.827 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:05:09.827 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:05:09.828 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:05:09.828 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:05:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:05:09.828 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:05:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:05:09.828 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:05:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:05:09.828 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:05:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:05:09.828 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:05:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:05:09.828 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:05:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:05:09.828 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:05:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:05:09.828 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:05:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:05:09.828 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:05:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:05:09.828 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:05:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:05:09.828 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:05:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:05:09.828 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:05:09.828 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:05:09.828 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:05:09.828 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:05:09.828 139328 ERROR oslo_service.periodic_task >2018-06-25 05:05:39.709 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:05:39.710 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:05:39.717 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:05:39.717 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:05:39.751 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:05:39.751 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:05:39.751 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:05:39.751 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:05:39.751 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:05:39.751 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:05:39.751 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:05:39.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:05:39.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:05:39.753 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:05:49.754 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:05:49.791 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:05:49.791 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:05:49.791 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:05:49.791 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:05:49.791 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:05:49.791 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:05:49.791 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:05:49.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:05:49.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:05:49.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:06:09.795 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:06:09.828 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:06:09.828 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:06:09.828 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:06:09.828 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:06:09.828 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:06:09.828 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:06:09.828 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:06:09.830 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:06:09.830 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:06:09.831 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:06:09.831 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:06:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:06:09.831 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:06:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:06:09.831 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:06:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:06:09.831 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:06:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:06:09.831 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:06:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:06:09.831 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:06:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:06:09.831 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:06:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:06:09.831 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:06:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:06:09.831 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:06:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:06:09.831 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:06:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:06:09.831 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:06:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:06:09.831 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:06:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:06:09.831 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:06:09.831 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:06:09.831 139328 ERROR oslo_service.periodic_task >2018-06-25 05:06:39.712 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:06:39.713 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:06:39.720 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:06:39.722 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:06:39.758 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:06:39.758 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:06:39.758 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:06:39.758 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:06:39.758 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:06:39.758 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:06:39.758 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:06:39.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:06:39.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:06:39.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:06:49.761 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:06:49.798 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:06:49.798 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:06:49.798 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:06:49.798 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:06:49.798 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:06:49.798 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:06:49.798 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:06:49.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:06:49.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:06:49.801 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:07:09.801 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:07:09.837 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:07:09.837 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:07:09.837 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:07:09.837 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:07:09.837 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:07:09.837 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:07:09.837 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:07:09.840 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:07:09.840 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.118 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:07:09.841 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:07:09.841 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:07:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:07:09.841 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:07:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:07:09.841 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:07:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:07:09.841 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:07:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:07:09.841 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:07:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:07:09.841 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:07:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:07:09.841 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:07:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:07:09.841 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:07:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:07:09.841 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:07:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:07:09.841 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:07:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:07:09.841 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:07:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:07:09.841 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:07:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:07:09.841 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:07:09.841 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:07:09.841 139328 ERROR oslo_service.periodic_task >2018-06-25 05:07:39.712 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:07:39.713 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:07:39.724 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:07:39.725 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:07:39.757 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:07:39.757 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:07:39.757 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:07:39.757 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:07:39.757 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:07:39.757 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:07:39.757 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:07:39.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:07:39.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:07:39.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:07:49.760 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:07:49.797 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:07:49.797 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:07:49.797 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:07:49.797 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:07:49.797 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:07:49.797 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:07:49.797 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:07:49.799 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:07:49.799 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:07:49.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:08:09.800 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:08:09.834 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:08:09.834 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:08:09.834 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:08:09.834 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:08:09.834 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:08:09.834 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:08:09.834 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:08:09.836 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:08:09.836 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:08:09.836 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:08:09.836 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:08:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:08:09.836 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:08:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:08:09.836 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:08:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:08:09.836 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:08:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:08:09.836 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:08:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:08:09.836 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:08:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:08:09.836 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:08:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:08:09.836 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:08:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:08:09.836 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:08:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:08:09.836 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:08:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:08:09.836 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:08:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:08:09.836 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:08:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:08:09.836 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:08:09.836 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:08:09.836 139328 ERROR oslo_service.periodic_task >2018-06-25 05:08:39.708 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:08:39.709 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:08:39.718 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:08:39.718 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:08:39.770 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:08:39.770 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:08:39.770 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:08:39.770 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:08:39.770 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:08:39.770 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:08:39.770 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:08:39.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:08:39.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.053 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:08:39.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:08:49.773 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:08:49.811 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:08:49.811 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:08:49.811 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:08:49.811 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:08:49.811 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:08:49.811 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:08:49.811 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:08:49.813 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:08:49.813 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.094 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:08:49.814 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:09:09.814 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:09:09.848 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:09:09.848 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:09:09.848 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:09:09.848 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:09:09.848 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:09:09.848 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:09:09.848 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:09:09.850 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:09:09.850 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.131 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:09:09.850 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:09:09.850 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:09:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:09:09.850 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:09:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:09:09.850 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:09:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:09:09.850 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:09:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:09:09.850 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:09:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:09:09.850 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:09:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:09:09.850 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:09:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:09:09.850 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:09:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:09:09.850 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:09:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:09:09.850 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:09:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:09:09.850 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:09:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:09:09.850 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:09:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:09:09.850 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:09:09.850 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:09:09.850 139328 ERROR oslo_service.periodic_task >2018-06-25 05:09:39.712 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:09:39.712 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:09:39.719 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:09:39.720 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:09:39.757 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:09:39.757 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:09:39.757 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:09:39.757 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:09:39.757 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:09:39.757 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:09:39.757 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:09:39.759 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:09:39.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:09:39.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:09:49.760 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:09:49.793 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:09:49.793 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:09:49.793 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:09:49.793 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:09:49.793 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:09:49.793 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:09:49.793 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:09:49.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:09:49.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:09:49.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:10:09.796 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:10:09.829 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:10:09.829 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:10:09.829 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:10:09.829 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:10:09.829 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:10:09.829 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:10:09.829 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:10:09.831 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:10:09.831 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:10:09.831 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:10:09.831 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:10:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:10:09.831 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:10:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:10:09.831 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:10:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:10:09.831 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:10:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:10:09.831 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:10:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:10:09.831 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:10:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:10:09.831 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:10:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:10:09.831 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:10:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:10:09.831 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:10:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:10:09.831 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:10:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:10:09.831 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:10:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:10:09.831 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:10:09.831 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:10:09.831 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:10:09.831 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:10:09.831 139328 ERROR oslo_service.periodic_task >2018-06-25 05:10:39.713 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:10:39.714 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:10:39.721 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:10:39.722 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:10:39.758 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:10:39.758 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:10:39.758 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:10:39.758 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:10:39.758 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:10:39.758 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:10:39.758 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:10:39.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:10:39.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:10:39.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:10:49.761 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:10:49.795 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:10:49.795 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:10:49.795 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:10:49.795 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:10:49.795 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:10:49.795 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:10:49.795 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:10:49.797 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:10:49.798 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:10:49.798 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:11:09.798 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:11:09.833 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:11:09.833 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:11:09.833 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:11:09.833 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:11:09.833 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:11:09.833 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:11:09.833 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:11:09.835 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:11:09.835 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:11:09.836 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:11:09.836 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:11:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:11:09.836 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:11:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:11:09.836 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:11:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:11:09.836 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:11:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:11:09.836 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:11:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:11:09.836 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:11:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:11:09.836 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:11:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:11:09.836 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:11:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:11:09.836 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:11:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:11:09.836 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:11:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:11:09.836 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:11:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:11:09.836 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:11:09.836 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:11:09.836 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:11:09.836 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:11:09.836 139328 ERROR oslo_service.periodic_task >2018-06-25 05:11:39.717 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:11:39.718 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:11:39.726 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:11:39.726 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:11:39.759 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:11:39.759 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:11:39.759 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:11:39.759 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:11:39.759 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:11:39.759 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:11:39.759 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:11:39.760 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:11:39.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:11:39.761 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:11:49.761 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:11:49.795 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:11:49.795 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:11:49.795 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:11:49.795 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:11:49.795 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:11:49.795 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:11:49.795 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:11:49.797 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:11:49.797 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:11:49.798 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:12:09.798 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:12:09.832 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:12:09.832 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:12:09.832 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:12:09.832 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:12:09.832 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:12:09.832 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:12:09.832 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:12:09.834 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:12:09.834 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.107 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:12:09.834 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:12:09.834 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:12:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:12:09.834 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:12:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:12:09.834 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:12:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:12:09.834 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:12:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:12:09.834 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:12:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:12:09.834 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:12:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:12:09.834 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:12:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:12:09.834 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:12:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:12:09.834 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:12:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:12:09.834 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:12:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:12:09.834 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:12:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:12:09.834 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:12:09.834 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:12:09.834 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:12:09.834 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:12:09.834 139328 ERROR oslo_service.periodic_task >2018-06-25 05:12:39.716 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:12:39.717 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:12:39.724 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:12:39.725 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:12:39.759 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:12:39.759 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:12:39.759 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:12:39.759 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:12:39.759 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:12:39.759 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:12:39.759 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:12:39.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:12:39.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:12:39.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:12:49.763 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:12:49.799 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:12:49.799 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:12:49.799 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:12:49.799 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:12:49.799 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:12:49.799 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:12:49.799 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:12:49.801 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:12:49.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:12:49.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:13:09.802 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:13:09.835 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:13:09.835 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:13:09.835 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:13:09.835 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:13:09.835 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:13:09.835 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:13:09.835 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:13:09.837 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:13:09.837 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:13:09.838 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:13:09.838 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:13:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:13:09.838 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:13:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:13:09.838 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:13:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:13:09.838 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:13:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:13:09.838 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:13:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:13:09.838 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:13:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:13:09.838 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:13:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:13:09.838 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:13:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:13:09.838 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:13:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:13:09.838 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:13:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:13:09.838 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:13:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:13:09.838 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:13:09.838 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:13:09.838 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:13:09.838 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:13:09.838 139328 ERROR oslo_service.periodic_task >2018-06-25 05:13:39.719 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:13:39.720 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:13:39.729 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:13:39.729 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:13:39.760 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:13:39.760 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:13:39.760 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:13:39.760 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:13:39.760 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:13:39.760 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:13:39.760 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:13:39.762 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:13:39.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.033 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:13:39.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:13:49.763 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:13:49.796 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:13:49.796 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:13:49.796 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:13:49.796 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:13:49.796 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:13:49.796 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:13:49.796 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:13:49.798 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:13:49.799 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.069 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:13:49.799 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:14:09.803 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:14:09.843 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:14:09.843 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:14:09.843 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:14:09.843 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:14:09.843 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:14:09.843 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:14:09.843 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:14:09.847 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:14:09.847 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:14:09.848 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:14:09.848 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:14:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:14:09.848 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:14:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:14:09.848 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:14:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:14:09.848 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:14:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:14:09.848 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:14:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:14:09.848 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:14:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:14:09.848 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:14:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:14:09.848 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:14:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:14:09.848 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:14:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:14:09.848 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:14:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:14:09.848 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:14:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:14:09.848 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:14:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:14:09.848 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:14:09.848 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:14:09.848 139328 ERROR oslo_service.periodic_task >2018-06-25 05:14:39.719 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:14:39.720 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:14:39.728 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:14:39.728 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:14:39.762 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:14:39.762 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:14:39.762 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:14:39.762 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:14:39.762 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:14:39.762 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:14:39.762 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:14:39.764 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:14:39.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:14:39.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:14:49.765 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:14:49.803 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:14:49.803 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:14:49.803 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:14:49.803 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:14:49.803 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:14:49.803 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:14:49.803 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:14:49.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:14:49.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:14:49.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:15:09.807 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:15:09.839 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:15:09.839 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:15:09.839 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:15:09.839 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:15:09.839 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:15:09.839 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:15:09.839 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:15:09.840 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:15:09.841 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:15:09.841 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:15:09.841 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:15:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:15:09.841 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:15:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:15:09.841 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:15:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:15:09.841 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:15:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:15:09.841 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:15:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:15:09.841 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:15:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:15:09.841 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:15:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:15:09.841 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:15:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:15:09.841 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:15:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:15:09.841 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:15:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:15:09.841 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:15:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:15:09.841 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:15:09.841 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:15:09.841 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:15:09.841 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:15:09.841 139328 ERROR oslo_service.periodic_task >2018-06-25 05:15:39.722 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:15:39.723 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:15:39.729 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:15:39.730 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:15:39.761 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:15:39.761 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:15:39.761 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:15:39.761 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:15:39.761 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:15:39.761 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:15:39.761 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:15:39.763 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:15:39.764 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:15:39.764 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:15:49.764 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:15:49.798 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:15:49.798 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:15:49.798 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:15:49.798 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:15:49.798 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:15:49.798 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:15:49.798 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:15:49.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:15:49.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.07 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:15:49.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:16:09.801 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:16:09.833 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:16:09.833 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:16:09.833 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:16:09.833 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:16:09.833 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:16:09.833 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:16:09.833 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:16:09.835 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:16:09.835 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.105 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:16:09.835 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:16:09.835 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:16:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:16:09.835 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:16:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:16:09.835 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:16:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:16:09.835 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:16:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:16:09.835 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:16:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:16:09.835 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:16:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:16:09.835 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:16:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:16:09.835 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:16:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:16:09.835 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:16:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:16:09.835 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:16:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:16:09.835 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:16:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:16:09.835 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:16:09.835 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:16:09.835 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:16:09.835 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:16:09.835 139328 ERROR oslo_service.periodic_task >2018-06-25 05:16:39.727 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:16:39.728 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:16:39.734 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:16:39.734 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:16:39.767 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:16:39.767 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:16:39.767 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:16:39.767 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:16:39.767 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:16:39.767 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:16:39.767 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:16:39.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:16:39.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:16:39.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:16:49.771 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:16:49.805 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:16:49.805 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:16:49.805 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:16:49.805 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:16:49.805 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:16:49.805 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:16:49.805 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:16:49.807 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:16:49.808 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:16:49.808 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:17:09.809 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:17:09.842 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:17:09.842 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:17:09.842 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:17:09.842 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:17:09.842 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:17:09.842 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:17:09.842 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:17:09.844 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:17:09.845 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:17:09.845 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:17:09.845 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:17:09.845 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:17:09.845 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:17:09.845 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:17:09.845 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:17:09.845 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:17:09.845 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:17:09.845 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:17:09.845 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:17:09.845 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:17:09.845 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:17:09.845 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:17:09.845 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:17:09.845 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:17:09.845 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:17:09.845 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:17:09.845 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:17:09.845 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:17:09.845 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:17:09.845 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:17:09.845 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:17:09.845 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:17:09.845 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:17:09.845 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:17:09.845 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:17:09.845 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:17:09.845 139328 ERROR oslo_service.periodic_task >2018-06-25 05:17:39.727 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:17:39.728 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:17:39.735 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:17:39.735 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:17:39.767 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:17:39.767 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:17:39.767 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:17:39.767 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:17:39.767 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:17:39.767 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:17:39.767 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:17:39.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:17:39.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.033 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:17:39.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:17:49.770 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:17:49.803 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:17:49.803 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:17:49.803 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:17:49.803 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:17:49.803 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:17:49.803 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:17:49.803 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:17:49.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:17:49.807 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:17:49.807 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:18:09.808 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:18:09.839 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:18:09.839 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:18:09.839 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:18:09.839 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:18:09.839 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:18:09.839 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:18:09.839 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:18:09.841 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:18:09.841 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.106 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:18:09.842 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:18:09.842 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:18:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:18:09.842 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:18:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:18:09.842 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:18:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:18:09.842 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:18:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:18:09.842 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:18:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:18:09.842 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:18:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:18:09.842 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:18:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:18:09.842 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:18:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:18:09.842 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:18:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:18:09.842 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:18:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:18:09.842 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:18:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:18:09.842 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:18:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:18:09.842 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:18:09.842 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:18:09.842 139328 ERROR oslo_service.periodic_task >2018-06-25 05:18:39.723 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:18:39.724 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:18:39.730 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:18:39.731 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:18:39.763 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:18:39.763 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:18:39.763 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:18:39.763 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:18:39.763 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:18:39.763 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:18:39.763 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:18:39.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:18:39.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:18:39.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:18:49.766 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:18:49.801 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:18:49.801 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:18:49.801 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:18:49.801 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:18:49.801 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:18:49.801 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:18:49.801 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:18:49.804 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:18:49.804 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:18:49.804 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:19:09.805 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:19:09.841 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:19:09.841 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:19:09.841 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:19:09.841 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:19:09.841 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:19:09.841 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:19:09.841 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:19:09.843 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:19:09.844 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:19:09.844 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:19:09.844 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:19:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:19:09.844 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:19:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:19:09.844 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:19:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:19:09.844 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:19:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:19:09.844 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:19:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:19:09.844 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:19:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:19:09.844 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:19:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:19:09.844 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:19:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:19:09.844 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:19:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:19:09.844 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:19:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:19:09.844 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:19:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:19:09.844 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:19:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:19:09.844 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:19:09.844 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:19:09.844 139328 ERROR oslo_service.periodic_task >2018-06-25 05:19:39.725 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:19:39.726 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:19:39.737 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:19:39.737 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:19:39.789 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:19:39.789 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:19:39.789 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:19:39.789 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:19:39.789 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:19:39.789 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:19:39.789 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:19:39.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:19:39.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.054 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:19:39.792 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:19:49.793 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:19:49.827 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:19:49.827 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:19:49.827 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:19:49.827 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:19:49.827 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:19:49.827 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:19:49.827 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:19:49.832 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:19:49.832 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.094 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:19:49.832 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:20:09.833 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:20:09.867 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:20:09.867 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:20:09.867 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:20:09.867 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:20:09.867 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:20:09.867 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:20:09.867 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:20:09.869 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:20:09.870 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.132 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:20:09.870 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:20:09.870 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:20:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:20:09.870 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:20:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:20:09.870 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:20:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:20:09.870 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:20:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:20:09.870 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:20:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:20:09.870 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:20:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:20:09.870 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:20:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:20:09.870 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:20:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:20:09.870 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:20:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:20:09.870 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:20:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:20:09.870 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:20:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:20:09.870 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:20:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:20:09.870 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:20:09.870 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:20:09.870 139328 ERROR oslo_service.periodic_task >2018-06-25 05:20:39.722 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:20:39.722 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:20:39.729 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:20:39.729 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:20:39.762 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:20:39.762 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:20:39.762 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:20:39.762 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:20:39.762 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:20:39.762 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:20:39.762 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:20:39.764 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:20:39.764 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:20:39.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:20:49.765 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:20:49.800 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:20:49.800 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:20:49.800 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:20:49.800 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:20:49.800 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:20:49.800 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:20:49.800 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:20:49.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:20:49.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:20:49.803 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:21:09.803 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:21:09.837 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:21:09.837 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:21:09.837 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:21:09.837 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:21:09.837 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:21:09.837 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:21:09.837 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:21:09.839 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:21:09.840 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:21:09.840 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:21:09.840 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:21:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:21:09.840 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:21:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:21:09.840 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:21:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:21:09.840 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:21:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:21:09.840 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:21:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:21:09.840 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:21:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:21:09.840 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:21:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:21:09.840 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:21:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:21:09.840 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:21:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:21:09.840 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:21:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:21:09.840 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:21:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:21:09.840 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:21:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:21:09.840 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:21:09.840 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:21:09.840 139328 ERROR oslo_service.periodic_task >2018-06-25 05:21:39.722 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:21:39.722 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:21:39.729 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:21:39.729 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:21:39.763 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:21:39.763 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:21:39.763 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:21:39.763 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:21:39.763 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:21:39.763 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:21:39.763 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:21:39.764 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:21:39.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:21:39.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:21:49.765 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:21:49.805 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:21:49.805 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:21:49.805 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:21:49.805 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:21:49.805 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:21:49.805 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:21:49.805 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:21:49.808 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:21:49.808 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:21:49.809 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:22:09.809 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:22:09.842 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:22:09.842 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:22:09.842 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:22:09.842 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:22:09.842 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:22:09.842 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:22:09.842 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:22:09.844 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:22:09.845 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:22:09.845 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:22:09.845 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:22:09.845 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:22:09.845 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:22:09.845 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:22:09.845 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:22:09.845 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:22:09.845 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:22:09.845 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:22:09.845 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:22:09.845 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:22:09.845 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:22:09.845 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:22:09.845 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:22:09.845 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:22:09.845 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:22:09.845 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:22:09.845 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:22:09.845 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:22:09.845 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:22:09.845 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:22:09.845 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:22:09.845 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:22:09.845 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:22:09.845 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:22:09.845 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:22:09.845 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:22:09.845 139328 ERROR oslo_service.periodic_task >2018-06-25 05:22:39.726 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:22:39.727 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:22:39.738 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:22:39.739 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:22:39.774 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:22:39.774 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:22:39.774 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:22:39.774 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:22:39.774 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:22:39.774 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:22:39.774 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:22:39.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:22:39.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:22:39.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:22:49.777 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:22:49.817 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:22:49.817 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:22:49.817 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:22:49.817 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:22:49.817 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:22:49.817 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:22:49.817 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:22:49.819 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:22:49.820 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.081 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:22:49.820 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:23:09.821 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:23:09.854 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:23:09.854 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:23:09.854 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:23:09.854 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:23:09.854 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:23:09.854 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:23:09.854 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:23:09.857 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:23:09.857 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.118 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:23:09.857 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:23:09.857 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:23:09.857 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:23:09.857 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:23:09.857 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:23:09.857 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:23:09.857 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:23:09.857 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:23:09.857 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:23:09.857 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:23:09.857 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:23:09.857 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:23:09.857 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:23:09.857 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:23:09.857 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:23:09.857 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:23:09.857 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:23:09.857 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:23:09.857 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:23:09.857 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:23:09.857 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:23:09.857 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:23:09.857 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:23:09.857 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:23:09.857 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:23:09.857 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:23:09.857 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:23:09.857 139328 ERROR oslo_service.periodic_task >2018-06-25 05:23:39.729 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:23:39.729 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:23:39.737 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:23:39.737 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:23:39.772 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:23:39.772 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:23:39.772 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:23:39.772 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:23:39.772 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:23:39.772 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:23:39.772 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:23:39.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:23:39.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:23:39.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:23:49.775 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:23:49.808 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:23:49.808 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:23:49.808 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:23:49.808 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:23:49.808 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:23:49.808 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:23:49.808 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:23:49.809 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:23:49.810 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:23:49.810 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:24:09.811 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:24:09.847 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:24:09.847 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:24:09.847 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:24:09.847 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:24:09.847 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:24:09.847 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:24:09.847 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:24:09.849 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:24:09.850 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:24:09.850 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:24:09.850 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:24:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:24:09.850 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:24:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:24:09.850 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:24:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:24:09.850 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:24:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:24:09.850 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:24:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:24:09.850 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:24:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:24:09.850 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:24:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:24:09.850 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:24:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:24:09.850 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:24:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:24:09.850 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:24:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:24:09.850 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:24:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:24:09.850 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:24:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:24:09.850 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:24:09.850 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:24:09.850 139328 ERROR oslo_service.periodic_task >2018-06-25 05:24:39.731 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:24:39.732 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:24:39.740 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:24:39.740 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:24:39.774 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:24:39.774 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:24:39.774 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:24:39.774 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:24:39.774 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:24:39.774 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:24:39.774 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:24:39.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:24:39.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:24:39.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:24:49.777 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:24:49.817 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:24:49.817 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:24:49.817 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:24:49.817 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:24:49.817 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:24:49.817 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:24:49.817 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:24:49.821 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:24:49.822 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.081 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:24:49.822 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:25:09.822 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:25:09.863 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:25:09.863 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:25:09.863 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:25:09.863 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:25:09.863 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:25:09.863 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:25:09.863 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:25:09.867 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:25:09.867 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.126 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:25:09.867 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:25:09.867 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:25:09.867 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:25:09.867 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:25:09.867 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:25:09.867 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:25:09.867 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:25:09.867 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:25:09.867 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:25:09.867 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:25:09.867 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:25:09.867 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:25:09.867 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:25:09.867 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:25:09.867 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:25:09.867 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:25:09.867 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:25:09.867 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:25:09.867 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:25:09.867 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:25:09.867 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:25:09.867 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:25:09.867 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:25:09.867 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:25:09.867 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:25:09.867 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:25:09.867 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:25:09.867 139328 ERROR oslo_service.periodic_task >2018-06-25 05:25:39.729 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:25:39.729 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:25:39.741 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:25:39.742 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:25:39.776 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:25:39.776 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:25:39.776 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:25:39.776 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:25:39.776 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:25:39.776 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:25:39.776 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:25:39.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:25:39.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:25:39.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:25:49.779 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:25:49.812 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:25:49.812 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:25:49.812 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:25:49.812 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:25:49.812 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:25:49.812 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:25:49.812 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:25:49.813 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:25:49.814 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:25:49.814 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:26:09.815 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:26:09.847 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:26:09.847 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:26:09.847 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:26:09.847 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:26:09.847 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:26:09.847 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:26:09.847 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:26:09.848 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:26:09.849 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.107 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:26:09.849 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:26:09.849 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:26:09.849 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:26:09.849 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:26:09.849 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:26:09.849 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:26:09.849 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:26:09.849 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:26:09.849 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:26:09.849 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:26:09.849 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:26:09.849 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:26:09.849 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:26:09.849 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:26:09.849 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:26:09.849 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:26:09.849 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:26:09.849 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:26:09.849 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:26:09.849 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:26:09.849 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:26:09.849 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:26:09.849 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:26:09.849 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:26:09.849 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:26:09.849 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:26:09.849 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:26:09.849 139328 ERROR oslo_service.periodic_task >2018-06-25 05:26:39.730 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:26:39.732 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:26:39.739 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:26:39.739 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:26:39.771 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:26:39.771 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:26:39.771 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:26:39.771 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:26:39.771 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:26:39.771 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:26:39.771 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:26:39.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:26:39.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:26:39.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:26:49.775 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:26:49.814 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:26:49.814 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:26:49.814 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:26:49.814 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:26:49.814 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:26:49.814 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:26:49.814 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:26:49.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:26:49.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:26:49.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:27:09.817 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:27:09.850 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:27:09.850 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:27:09.850 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:27:09.850 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:27:09.850 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:27:09.850 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:27:09.850 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:27:09.852 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:27:09.852 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:27:09.853 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:27:09.853 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:27:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:27:09.853 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:27:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:27:09.853 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:27:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:27:09.853 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:27:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:27:09.853 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:27:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:27:09.853 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:27:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:27:09.853 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:27:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:27:09.853 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:27:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:27:09.853 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:27:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:27:09.853 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:27:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:27:09.853 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:27:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:27:09.853 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:27:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:27:09.853 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:27:09.853 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:27:09.853 139328 ERROR oslo_service.periodic_task >2018-06-25 05:27:39.734 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:27:39.735 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:27:39.743 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:27:39.743 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:27:39.777 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:27:39.777 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:27:39.777 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:27:39.777 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:27:39.777 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:27:39.777 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:27:39.777 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:27:39.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:27:39.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:27:39.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:27:49.780 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:27:49.814 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:27:49.814 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:27:49.814 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:27:49.814 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:27:49.814 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:27:49.814 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:27:49.814 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:27:49.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:27:49.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:27:49.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:28:09.818 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:28:09.852 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:28:09.852 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:28:09.852 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:28:09.852 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:28:09.852 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:28:09.852 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:28:09.852 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:28:09.854 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:28:09.855 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:28:09.855 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:28:09.855 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:28:09.855 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:28:09.855 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:28:09.855 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:28:09.855 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:28:09.855 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:28:09.855 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:28:09.855 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:28:09.855 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:28:09.855 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:28:09.855 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:28:09.855 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:28:09.855 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:28:09.855 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:28:09.855 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:28:09.855 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:28:09.855 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:28:09.855 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:28:09.855 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:28:09.855 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:28:09.855 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:28:09.855 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:28:09.855 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:28:09.855 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:28:09.855 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:28:09.855 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:28:09.855 139328 ERROR oslo_service.periodic_task >2018-06-25 05:28:39.736 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:28:39.737 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:28:39.744 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:28:39.745 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:28:39.778 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:28:39.778 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:28:39.778 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:28:39.778 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:28:39.778 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:28:39.778 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:28:39.778 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:28:39.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:28:39.781 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:28:39.781 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:28:49.782 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:28:49.814 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:28:49.814 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:28:49.814 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:28:49.814 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:28:49.814 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:28:49.814 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:28:49.814 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:28:49.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:28:49.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.07 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:28:49.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:29:09.817 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:29:09.855 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:29:09.855 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:29:09.855 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:29:09.855 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:29:09.855 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:29:09.855 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:29:09.855 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:29:09.857 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:29:09.858 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:29:09.859 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:29:09.859 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:29:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:29:09.859 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:29:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:29:09.859 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:29:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:29:09.859 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:29:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:29:09.859 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:29:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:29:09.859 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:29:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:29:09.859 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:29:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:29:09.859 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:29:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:29:09.859 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:29:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:29:09.859 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:29:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:29:09.859 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:29:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:29:09.859 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:29:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:29:09.859 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:29:09.859 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:29:09.859 139328 ERROR oslo_service.periodic_task >2018-06-25 05:29:39.740 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:29:39.740 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:29:39.748 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:29:39.749 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:29:39.785 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:29:39.785 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:29:39.785 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:29:39.785 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:29:39.785 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:29:39.785 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:29:39.785 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:29:39.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:29:39.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:29:39.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:29:49.788 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:29:49.823 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:29:49.823 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:29:49.823 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:29:49.823 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:29:49.823 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:29:49.823 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:29:49.823 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:29:49.825 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:29:49.825 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:29:49.826 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:30:09.826 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:30:09.866 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:30:09.866 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:30:09.866 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:30:09.866 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:30:09.866 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:30:09.866 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:30:09.866 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:30:09.869 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:30:09.869 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.12 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:30:09.870 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:30:09.870 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:30:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:30:09.870 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:30:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:30:09.870 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:30:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:30:09.870 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:30:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:30:09.870 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:30:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:30:09.870 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:30:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:30:09.870 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:30:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:30:09.870 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:30:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:30:09.870 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:30:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:30:09.870 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:30:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:30:09.870 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:30:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:30:09.870 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:30:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:30:09.870 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:30:09.870 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:30:09.870 139328 ERROR oslo_service.periodic_task >2018-06-25 05:30:39.742 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:30:39.742 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:30:39.748 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:30:39.749 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:30:39.782 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:30:39.782 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:30:39.782 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:30:39.782 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:30:39.782 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:30:39.782 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:30:39.782 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:30:39.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:30:39.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:30:39.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:30:49.785 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:30:49.820 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:30:49.820 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:30:49.820 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:30:49.820 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:30:49.820 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:30:49.820 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:30:49.820 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:30:49.822 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:30:49.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:30:49.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:31:09.823 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:31:09.857 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:31:09.857 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:31:09.857 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:31:09.857 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:31:09.857 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:31:09.857 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:31:09.857 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:31:09.858 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:31:09.859 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:31:09.859 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:31:09.859 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:31:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:31:09.859 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:31:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:31:09.859 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:31:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:31:09.859 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:31:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:31:09.859 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:31:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:31:09.859 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:31:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:31:09.859 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:31:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:31:09.859 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:31:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:31:09.859 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:31:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:31:09.859 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:31:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:31:09.859 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:31:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:31:09.859 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:31:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:31:09.859 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:31:09.859 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:31:09.859 139328 ERROR oslo_service.periodic_task >2018-06-25 05:31:39.740 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:31:39.741 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:31:39.748 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:31:39.749 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:31:39.782 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:31:39.782 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:31:39.782 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:31:39.782 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:31:39.782 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:31:39.782 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:31:39.782 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:31:39.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:31:39.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:31:39.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:31:49.785 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:31:49.818 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:31:49.818 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:31:49.818 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:31:49.818 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:31:49.818 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:31:49.818 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:31:49.818 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:31:49.820 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:31:49.821 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:31:49.821 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:32:09.821 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:32:09.858 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:32:09.858 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:32:09.858 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:32:09.858 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:32:09.858 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:32:09.858 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:32:09.858 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:32:09.860 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:32:09.861 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:32:09.861 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:32:09.861 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:32:09.861 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:32:09.861 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:32:09.861 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:32:09.861 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:32:09.861 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:32:09.861 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:32:09.861 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:32:09.861 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:32:09.861 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:32:09.861 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:32:09.861 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:32:09.861 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:32:09.861 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:32:09.861 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:32:09.861 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:32:09.861 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:32:09.861 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:32:09.861 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:32:09.861 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:32:09.861 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:32:09.861 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:32:09.861 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:32:09.861 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:32:09.861 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:32:09.861 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:32:09.861 139328 ERROR oslo_service.periodic_task >2018-06-25 05:32:39.743 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:32:39.743 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:32:39.750 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:32:39.751 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:32:39.783 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:32:39.783 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:32:39.783 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:32:39.783 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:32:39.783 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:32:39.783 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:32:39.783 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:32:39.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:32:39.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:32:39.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:32:49.786 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:32:49.820 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:32:49.820 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:32:49.820 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:32:49.820 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:32:49.820 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:32:49.820 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:32:49.820 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:32:49.822 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:32:49.822 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:32:49.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:33:09.823 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:33:09.856 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:33:09.856 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:33:09.856 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:33:09.856 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:33:09.856 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:33:09.856 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:33:09.856 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:33:09.859 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:33:09.859 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:33:09.860 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:33:09.860 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:33:09.860 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:33:09.860 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:33:09.860 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:33:09.860 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:33:09.860 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:33:09.860 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:33:09.860 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:33:09.860 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:33:09.860 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:33:09.860 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:33:09.860 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:33:09.860 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:33:09.860 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:33:09.860 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:33:09.860 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:33:09.860 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:33:09.860 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:33:09.860 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:33:09.860 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:33:09.860 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:33:09.860 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:33:09.860 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:33:09.860 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:33:09.860 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:33:09.860 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:33:09.860 139328 ERROR oslo_service.periodic_task >2018-06-25 05:33:39.741 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:33:39.742 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:34:39.739 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:34:39.740 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:34:39.748 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:34:39.749 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:34:39.785 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:34:39.785 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:34:39.785 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:34:39.785 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:34:39.785 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:34:39.785 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:34:39.785 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:34:39.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:34:39.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:34:39.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:34:49.789 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:34:49.822 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:34:49.822 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:34:49.822 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:34:49.822 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:34:49.822 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:34:49.822 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:34:49.822 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:34:49.824 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:34:49.825 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:34:49.825 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:35:09.826 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:35:09.857 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:35:09.857 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:35:09.857 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:35:09.857 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:35:09.857 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:35:09.857 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:35:09.857 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:35:09.859 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:35:09.859 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:35:09.860 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:35:09.860 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:35:09.860 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:35:09.860 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:35:09.860 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:35:09.860 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:35:09.860 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:35:09.860 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:35:09.860 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:35:09.860 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:35:09.860 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:35:09.860 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:35:09.860 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:35:09.860 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:35:09.860 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:35:09.860 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:35:09.860 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:35:09.860 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:35:09.860 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:35:09.860 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:35:09.860 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:35:09.860 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:35:09.860 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:35:09.860 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:35:09.860 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:35:09.860 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:35:09.860 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:35:09.860 139328 ERROR oslo_service.periodic_task >2018-06-25 05:35:39.741 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:35:39.742 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:35:39.749 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:35:39.749 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:35:39.782 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:35:39.782 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:35:39.782 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:35:39.782 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:35:39.782 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:35:39.782 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:35:39.782 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:35:39.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:35:39.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:35:39.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:35:49.785 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:35:49.823 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:35:49.823 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:35:49.823 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:35:49.823 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:35:49.823 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:35:49.823 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:35:49.823 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:35:49.826 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:35:49.827 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:35:49.827 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:36:09.828 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:36:09.865 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:36:09.865 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:36:09.865 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:36:09.865 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:36:09.865 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:36:09.865 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:36:09.865 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:36:09.867 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:36:09.867 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:36:09.868 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:36:09.868 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:36:09.868 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:36:09.868 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:36:09.868 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:36:09.868 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:36:09.868 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:36:09.868 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:36:09.868 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:36:09.868 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:36:09.868 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:36:09.868 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:36:09.868 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:36:09.868 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:36:09.868 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:36:09.868 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:36:09.868 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:36:09.868 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:36:09.868 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:36:09.868 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:36:09.868 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:36:09.868 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:36:09.868 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:36:09.868 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:36:09.868 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:36:09.868 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:36:09.868 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:36:09.868 139328 ERROR oslo_service.periodic_task >2018-06-25 05:36:39.739 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:36:39.740 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:36:39.746 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:36:39.747 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:36:39.780 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:36:39.780 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:36:39.780 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:36:39.780 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:36:39.780 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:36:39.780 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:36:39.780 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:36:39.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:36:39.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:36:39.783 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:36:49.783 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:36:49.819 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:36:49.819 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:36:49.819 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:36:49.819 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:36:49.819 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:36:49.819 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:36:49.819 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:36:49.821 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:36:49.821 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:36:49.822 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:37:09.822 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:37:09.856 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:37:09.856 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:37:09.856 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:37:09.856 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:37:09.856 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:37:09.856 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:37:09.856 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:37:09.858 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:37:09.859 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:37:09.859 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:37:09.859 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:37:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:37:09.859 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:37:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:37:09.859 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:37:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:37:09.859 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:37:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:37:09.859 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:37:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:37:09.859 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:37:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:37:09.859 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:37:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:37:09.859 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:37:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:37:09.859 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:37:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:37:09.859 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:37:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:37:09.859 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:37:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:37:09.859 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:37:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:37:09.859 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:37:09.859 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:37:09.859 139328 ERROR oslo_service.periodic_task >2018-06-25 05:37:39.741 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:37:39.741 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:37:39.750 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:37:39.750 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:37:39.788 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:37:39.788 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:37:39.788 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:37:39.788 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:37:39.788 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:37:39.788 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:37:39.788 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:37:39.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:37:39.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:37:39.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:37:49.791 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:37:49.826 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:37:49.826 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:37:49.826 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:37:49.826 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:37:49.826 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:37:49.826 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:37:49.826 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:37:49.828 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:37:49.829 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:37:49.829 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:38:09.830 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:38:09.863 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:38:09.863 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:38:09.863 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:38:09.863 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:38:09.863 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:38:09.863 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:38:09.863 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:38:09.866 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:38:09.866 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:38:09.866 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:38:09.866 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:38:09.866 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:38:09.866 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:38:09.866 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:38:09.866 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:38:09.866 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:38:09.866 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:38:09.866 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:38:09.866 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:38:09.866 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:38:09.866 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:38:09.866 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:38:09.866 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:38:09.866 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:38:09.866 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:38:09.866 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:38:09.866 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:38:09.866 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:38:09.866 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:38:09.866 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:38:09.866 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:38:09.866 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:38:09.866 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:38:09.866 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:38:09.866 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:38:09.866 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:38:09.866 139328 ERROR oslo_service.periodic_task >2018-06-25 05:38:39.737 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:38:39.738 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:38:39.745 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:38:39.745 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:38:39.777 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:38:39.777 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:38:39.777 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:38:39.777 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:38:39.777 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:38:39.777 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:38:39.777 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:38:39.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:38:39.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:38:39.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:38:49.781 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:38:49.814 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:38:49.814 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:38:49.814 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:38:49.814 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:38:49.814 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:38:49.814 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:38:49.814 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:38:49.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:38:49.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:38:49.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:39:09.817 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:39:09.851 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:39:09.851 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:39:09.851 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:39:09.851 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:39:09.851 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:39:09.851 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:39:09.851 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:39:09.853 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:39:09.853 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.107 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:39:09.853 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:39:09.853 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:39:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:39:09.853 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:39:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:39:09.853 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:39:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:39:09.853 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:39:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:39:09.853 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:39:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:39:09.853 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:39:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:39:09.853 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:39:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:39:09.853 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:39:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:39:09.853 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:39:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:39:09.853 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:39:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:39:09.853 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:39:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:39:09.853 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:39:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:39:09.853 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:39:09.853 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:39:09.853 139328 ERROR oslo_service.periodic_task >2018-06-25 05:39:39.735 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:39:39.735 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:39:39.745 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:39:39.746 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:39:39.784 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:39:39.784 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:39:39.784 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:39:39.784 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:39:39.784 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:39:39.784 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:39:39.784 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:39:39.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:39:39.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.041 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:39:39.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:39:49.788 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:39:49.821 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:39:49.821 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:39:49.821 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:39:49.821 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:39:49.821 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:39:49.821 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:39:49.821 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:39:49.824 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:39:49.824 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:39:49.825 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:40:09.825 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:40:09.857 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:40:09.857 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:40:09.857 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:40:09.857 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:40:09.857 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:40:09.857 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:40:09.857 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:40:09.859 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:40:09.859 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:40:09.859 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:40:09.859 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:40:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:40:09.859 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:40:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:40:09.859 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:40:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:40:09.859 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:40:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:40:09.859 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:40:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:40:09.859 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:40:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:40:09.859 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:40:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:40:09.859 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:40:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:40:09.859 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:40:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:40:09.859 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:40:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:40:09.859 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:40:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:40:09.859 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:40:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:40:09.859 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:40:09.859 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:40:09.859 139328 ERROR oslo_service.periodic_task >2018-06-25 05:40:39.731 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:40:39.732 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:40:39.739 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:40:39.740 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:40:39.777 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:40:39.777 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:40:39.777 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:40:39.777 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:40:39.777 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:40:39.777 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:40:39.777 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:40:39.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:40:39.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:40:39.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:40:49.780 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:40:49.815 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:40:49.815 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:40:49.815 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:40:49.815 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:40:49.815 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:40:49.815 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:40:49.815 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:40:49.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:40:49.818 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:40:49.818 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:41:09.819 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:41:09.850 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:41:09.850 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:41:09.850 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:41:09.850 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:41:09.850 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:41:09.850 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:41:09.850 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:41:09.852 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:41:09.853 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:41:09.853 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:41:09.853 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:41:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:41:09.853 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:41:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:41:09.853 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:41:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:41:09.853 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:41:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:41:09.853 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:41:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:41:09.853 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:41:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:41:09.853 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:41:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:41:09.853 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:41:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:41:09.853 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:41:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:41:09.853 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:41:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:41:09.853 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:41:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:41:09.853 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:41:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:41:09.853 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:41:09.853 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:41:09.853 139328 ERROR oslo_service.periodic_task >2018-06-25 05:41:39.734 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:41:39.735 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:41:39.741 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:41:39.742 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:41:39.775 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:41:39.775 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:41:39.775 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:41:39.775 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:41:39.775 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:41:39.775 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:41:39.775 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:41:39.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:41:39.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:41:39.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:41:49.778 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:41:49.813 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:41:49.813 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:41:49.813 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:41:49.813 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:41:49.813 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:41:49.813 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:41:49.813 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:41:49.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:41:49.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:41:49.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:42:09.817 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:42:09.850 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:42:09.850 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:42:09.850 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:42:09.850 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:42:09.850 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:42:09.850 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:42:09.850 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:42:09.852 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:42:09.853 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:42:09.853 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:42:09.853 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:42:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:42:09.853 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:42:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:42:09.853 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:42:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:42:09.853 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:42:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:42:09.853 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:42:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:42:09.853 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:42:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:42:09.853 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:42:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:42:09.853 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:42:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:42:09.853 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:42:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:42:09.853 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:42:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:42:09.853 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:42:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:42:09.853 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:42:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:42:09.853 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:42:09.853 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:42:09.853 139328 ERROR oslo_service.periodic_task >2018-06-25 05:42:39.735 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:42:39.736 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:42:39.743 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:42:39.743 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:42:39.777 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:42:39.777 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:42:39.777 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:42:39.777 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:42:39.777 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:42:39.777 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:42:39.777 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:42:39.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:42:39.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:42:39.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:42:49.780 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:42:49.814 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:42:49.814 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:42:49.814 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:42:49.814 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:42:49.814 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:42:49.814 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:42:49.814 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:42:49.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:42:49.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:42:49.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:43:09.817 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:43:09.851 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:43:09.851 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:43:09.851 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:43:09.851 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:43:09.851 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:43:09.851 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:43:09.851 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:43:09.853 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:43:09.854 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:43:09.854 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:43:09.854 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:43:09.854 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:43:09.854 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:43:09.854 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:43:09.854 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:43:09.854 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:43:09.854 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:43:09.854 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:43:09.854 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:43:09.854 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:43:09.854 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:43:09.854 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:43:09.854 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:43:09.854 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:43:09.854 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:43:09.854 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:43:09.854 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:43:09.854 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:43:09.854 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:43:09.854 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:43:09.854 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:43:09.854 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:43:09.854 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:43:09.854 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:43:09.854 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:43:09.854 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:43:09.854 139328 ERROR oslo_service.periodic_task >2018-06-25 05:43:39.735 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:43:39.736 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:43:39.743 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:43:39.743 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:43:39.775 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:43:39.775 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:43:39.775 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:43:39.775 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:43:39.775 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:43:39.775 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:43:39.775 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:43:39.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:43:39.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.033 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:43:39.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:43:49.777 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:43:49.814 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:43:49.814 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:43:49.814 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:43:49.814 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:43:49.814 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:43:49.814 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:43:49.814 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:43:49.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:43:49.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:43:49.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:44:09.818 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:44:09.854 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:44:09.854 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:44:09.854 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:44:09.854 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:44:09.854 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:44:09.854 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:44:09.854 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:44:09.856 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:44:09.857 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:44:09.857 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:44:09.857 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:44:09.857 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:44:09.857 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:44:09.857 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:44:09.857 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:44:09.857 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:44:09.857 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:44:09.857 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:44:09.857 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:44:09.857 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:44:09.857 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:44:09.857 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:44:09.857 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:44:09.857 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:44:09.857 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:44:09.857 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:44:09.857 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:44:09.857 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:44:09.857 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:44:09.857 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:44:09.857 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:44:09.857 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:44:09.857 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:44:09.857 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:44:09.857 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:44:09.857 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:44:09.857 139328 ERROR oslo_service.periodic_task >2018-06-25 05:44:39.738 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:44:39.739 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:44:39.746 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:44:39.747 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:44:39.779 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:44:39.779 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:44:39.779 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:44:39.779 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:44:39.779 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:44:39.779 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:44:39.779 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:44:39.781 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:44:39.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:44:39.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:44:49.782 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:44:49.818 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:44:49.818 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:44:49.818 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:44:49.818 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:44:49.818 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:44:49.818 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:44:49.818 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:44:49.820 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:44:49.820 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:44:49.821 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:45:09.821 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:45:09.855 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:45:09.855 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:45:09.855 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:45:09.855 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:45:09.855 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:45:09.855 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:45:09.855 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:45:09.857 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:45:09.858 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:45:09.858 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:45:09.858 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:45:09.858 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:45:09.858 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:45:09.858 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:45:09.858 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:45:09.858 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:45:09.858 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:45:09.858 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:45:09.858 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:45:09.858 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:45:09.858 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:45:09.858 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:45:09.858 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:45:09.858 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:45:09.858 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:45:09.858 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:45:09.858 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:45:09.858 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:45:09.858 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:45:09.858 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:45:09.858 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:45:09.858 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:45:09.858 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:45:09.858 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:45:09.858 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:45:09.858 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:45:09.858 139328 ERROR oslo_service.periodic_task >2018-06-25 05:45:39.739 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:45:39.740 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:45:39.752 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:45:39.752 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:45:39.788 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:45:39.788 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:45:39.788 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:45:39.788 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:45:39.788 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:45:39.788 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:45:39.788 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:45:39.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:45:39.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:45:39.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:45:49.791 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:45:49.833 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:45:49.833 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:45:49.833 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:45:49.833 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:45:49.833 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:45:49.833 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:45:49.833 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:45:49.835 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:45:49.835 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.082 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:45:49.836 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:46:09.836 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:46:09.875 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:46:09.875 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:46:09.875 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:46:09.875 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:46:09.875 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:46:09.875 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:46:09.875 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:46:09.877 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:46:09.878 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.125 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:46:09.878 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:46:09.878 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:46:09.878 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:46:09.878 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:46:09.878 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:46:09.878 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:46:09.878 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:46:09.878 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:46:09.878 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:46:09.878 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:46:09.878 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:46:09.878 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:46:09.878 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:46:09.878 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:46:09.878 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:46:09.878 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:46:09.878 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:46:09.878 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:46:09.878 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:46:09.878 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:46:09.878 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:46:09.878 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:46:09.878 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:46:09.878 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:46:09.878 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:46:09.878 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:46:09.878 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:46:09.878 139328 ERROR oslo_service.periodic_task >2018-06-25 05:46:39.739 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:46:39.740 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:46:39.749 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:46:39.750 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:46:39.798 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:46:39.798 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:46:39.798 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:46:39.798 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:46:39.798 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:46:39.798 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:46:39.798 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:46:39.800 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:46:39.801 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.051 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:46:39.801 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:46:49.802 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:46:49.838 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:46:49.838 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:46:49.838 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:46:49.838 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:46:49.838 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:46:49.838 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:46:49.838 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:46:49.840 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:46:49.840 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.09 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:46:49.841 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:47:09.841 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:47:09.883 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:47:09.883 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:47:09.883 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:47:09.883 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:47:09.883 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:47:09.883 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:47:09.883 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:47:09.886 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:47:09.886 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.137 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:47:09.887 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:47:09.887 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:47:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:47:09.887 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:47:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:47:09.887 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:47:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:47:09.887 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:47:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:47:09.887 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:47:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:47:09.887 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:47:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:47:09.887 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:47:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:47:09.887 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:47:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:47:09.887 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:47:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:47:09.887 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:47:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:47:09.887 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:47:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:47:09.887 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:47:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:47:09.887 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:47:09.887 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:47:09.887 139328 ERROR oslo_service.periodic_task >2018-06-25 05:47:39.738 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:47:39.739 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:47:39.745 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:47:39.746 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:47:39.780 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:47:39.780 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:47:39.780 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:47:39.780 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:47:39.780 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:47:39.780 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:47:39.780 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:47:39.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:47:39.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:47:39.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:47:49.783 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:47:49.820 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:47:49.820 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:47:49.820 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:47:49.820 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:47:49.820 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:47:49.820 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:47:49.820 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:47:49.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:47:49.824 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:47:49.824 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:48:09.824 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:48:09.860 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:48:09.860 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:48:09.860 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:48:09.860 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:48:09.860 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:48:09.860 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:48:09.860 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:48:09.862 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:48:09.862 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:48:09.863 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:48:09.863 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:48:09.863 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:48:09.863 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:48:09.863 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:48:09.863 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:48:09.863 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:48:09.863 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:48:09.863 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:48:09.863 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:48:09.863 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:48:09.863 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:48:09.863 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:48:09.863 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:48:09.863 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:48:09.863 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:48:09.863 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:48:09.863 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:48:09.863 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:48:09.863 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:48:09.863 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:48:09.863 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:48:09.863 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:48:09.863 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:48:09.863 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:48:09.863 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:48:09.863 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:48:09.863 139328 ERROR oslo_service.periodic_task >2018-06-25 05:48:39.734 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:48:39.735 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:48:39.742 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:48:39.742 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:48:39.774 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:48:39.774 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:48:39.774 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:48:39.774 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:48:39.774 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:48:39.774 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:48:39.774 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:48:39.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:48:39.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:48:39.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:48:49.778 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:48:49.814 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:48:49.814 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:48:49.814 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:48:49.814 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:48:49.814 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:48:49.814 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:48:49.814 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:48:49.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:48:49.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:48:49.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:49:09.817 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:49:09.851 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:49:09.851 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:49:09.851 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:49:09.851 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:49:09.851 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:49:09.851 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:49:09.851 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:49:09.853 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:49:09.853 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:49:09.853 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:49:09.853 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:49:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:49:09.853 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:49:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:49:09.853 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:49:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:49:09.853 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:49:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:49:09.853 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:49:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:49:09.853 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:49:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:49:09.853 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:49:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:49:09.853 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:49:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:49:09.853 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:49:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:49:09.853 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:49:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:49:09.853 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:49:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:49:09.853 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:49:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:49:09.853 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:49:09.853 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:49:09.853 139328 ERROR oslo_service.periodic_task >2018-06-25 05:49:39.735 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:49:39.736 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:49:39.742 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:49:39.743 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:49:39.776 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:49:39.776 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:49:39.776 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:49:39.776 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:49:39.776 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:49:39.776 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:49:39.776 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:49:39.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:49:39.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:49:39.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:49:49.779 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:49:49.814 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:49:49.814 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:49:49.814 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:49:49.814 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:49:49.814 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:49:49.814 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:49:49.814 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:49:49.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:49:49.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:49:49.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:50:09.817 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:50:09.849 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:50:09.849 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:50:09.849 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:50:09.849 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:50:09.849 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:50:09.849 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:50:09.849 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:50:09.850 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:50:09.851 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:50:09.851 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:50:09.851 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:50:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:50:09.851 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:50:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:50:09.851 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:50:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:50:09.851 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:50:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:50:09.851 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:50:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:50:09.851 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:50:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:50:09.851 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:50:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:50:09.851 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:50:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:50:09.851 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:50:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:50:09.851 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:50:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:50:09.851 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:50:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:50:09.851 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:50:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:50:09.851 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:50:09.851 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:50:09.851 139328 ERROR oslo_service.periodic_task >2018-06-25 05:50:39.732 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:50:39.733 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:50:39.740 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:50:39.741 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:50:39.774 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:50:39.774 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:50:39.774 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:50:39.774 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:50:39.774 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:50:39.774 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:50:39.774 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:50:39.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:50:39.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:50:39.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:50:49.778 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:50:49.812 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:50:49.812 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:50:49.812 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:50:49.812 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:50:49.812 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:50:49.812 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:50:49.812 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:50:49.814 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:50:49.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:50:49.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:51:09.815 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:51:09.847 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:51:09.847 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:51:09.847 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:51:09.847 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:51:09.847 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:51:09.847 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:51:09.847 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:51:09.849 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:51:09.849 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:51:09.850 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:51:09.850 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:51:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:51:09.850 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:51:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:51:09.850 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:51:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:51:09.850 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:51:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:51:09.850 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:51:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:51:09.850 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:51:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:51:09.850 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:51:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:51:09.850 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:51:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:51:09.850 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:51:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:51:09.850 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:51:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:51:09.850 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:51:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:51:09.850 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:51:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:51:09.850 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:51:09.850 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:51:09.850 139328 ERROR oslo_service.periodic_task >2018-06-25 05:51:39.731 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:51:39.731 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:51:39.739 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:51:39.739 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:51:39.773 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:51:39.773 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:51:39.773 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:51:39.773 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:51:39.773 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:51:39.773 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:51:39.773 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:51:39.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:51:39.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:51:39.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:51:49.776 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:51:49.816 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:51:49.816 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:51:49.816 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:51:49.816 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:51:49.816 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:51:49.816 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:51:49.816 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:51:49.818 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:51:49.818 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:51:49.818 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:52:09.819 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:52:09.851 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:52:09.851 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:52:09.851 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:52:09.851 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:52:09.851 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:52:09.851 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:52:09.851 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:52:09.853 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:52:09.853 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:52:09.854 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:52:09.854 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:52:09.854 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:52:09.854 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:52:09.854 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:52:09.854 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:52:09.854 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:52:09.854 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:52:09.854 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:52:09.854 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:52:09.854 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:52:09.854 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:52:09.854 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:52:09.854 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:52:09.854 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:52:09.854 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:52:09.854 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:52:09.854 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:52:09.854 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:52:09.854 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:52:09.854 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:52:09.854 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:52:09.854 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:52:09.854 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:52:09.854 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:52:09.854 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:52:09.854 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:52:09.854 139328 ERROR oslo_service.periodic_task >2018-06-25 05:52:39.735 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:52:39.736 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:52:39.768 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:52:39.768 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:52:39.768 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:52:39.768 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:52:39.768 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:52:39.768 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:52:39.768 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:52:39.770 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:52:39.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:52:39.771 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:52:49.772 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:52:49.810 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:52:49.810 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:52:49.810 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:52:49.810 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:52:49.810 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:52:49.810 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:52:49.810 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:52:49.812 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:52:49.812 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:52:49.812 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:53:09.813 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:53:09.849 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:53:09.849 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:53:09.849 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:53:09.849 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:53:09.849 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:53:09.849 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:53:09.849 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:53:09.851 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:53:09.852 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:53:09.852 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:53:09.852 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:53:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:53:09.852 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:53:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:53:09.852 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:53:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:53:09.852 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:53:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:53:09.852 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:53:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:53:09.852 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:53:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:53:09.852 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:53:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:53:09.852 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:53:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:53:09.852 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:53:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:53:09.852 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:53:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:53:09.852 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:53:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:53:09.852 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:53:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:53:09.852 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:53:09.852 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:53:09.852 139328 ERROR oslo_service.periodic_task >2018-06-25 05:53:39.734 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:53:39.734 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:53:39.742 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:53:39.743 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:53:39.783 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:53:39.783 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:53:39.783 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:53:39.783 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:53:39.783 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:53:39.783 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:53:39.783 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:53:39.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:53:39.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.044 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:53:39.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:53:49.788 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:53:49.821 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:53:49.821 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:53:49.821 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:53:49.821 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:53:49.821 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:53:49.821 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:53:49.821 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:53:49.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:53:49.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.081 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:53:49.824 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:54:09.824 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:54:09.861 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:54:09.861 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:54:09.861 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:54:09.861 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:54:09.861 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:54:09.861 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:54:09.861 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:54:09.863 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:54:09.864 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.121 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:54:09.864 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:54:09.864 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:54:09.864 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:54:09.864 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:54:09.864 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:54:09.864 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:54:09.864 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:54:09.864 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:54:09.864 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:54:09.864 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:54:09.864 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:54:09.864 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:54:09.864 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:54:09.864 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:54:09.864 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:54:09.864 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:54:09.864 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:54:09.864 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:54:09.864 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:54:09.864 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:54:09.864 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:54:09.864 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:54:09.864 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:54:09.864 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:54:09.864 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:54:09.864 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:54:09.864 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:54:09.864 139328 ERROR oslo_service.periodic_task >2018-06-25 05:54:39.735 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:54:39.736 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:54:39.743 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:54:39.743 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:54:39.777 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:54:39.777 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:54:39.777 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:54:39.777 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:54:39.777 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:54:39.777 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:54:39.777 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:54:39.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:54:39.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:54:39.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:54:49.780 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:54:49.814 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:54:49.814 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:54:49.814 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:54:49.814 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:54:49.814 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:54:49.814 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:54:49.814 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:54:49.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:54:49.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:54:49.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:55:09.818 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:55:09.854 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:55:09.854 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:55:09.854 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:55:09.854 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:55:09.854 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:55:09.854 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:55:09.854 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:55:09.858 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:55:09.859 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:55:09.859 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:55:09.859 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:55:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:55:09.859 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:55:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:55:09.859 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:55:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:55:09.859 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:55:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:55:09.859 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:55:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:55:09.859 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:55:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:55:09.859 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:55:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:55:09.859 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:55:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:55:09.859 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:55:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:55:09.859 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:55:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:55:09.859 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:55:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:55:09.859 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:55:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:55:09.859 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:55:09.859 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:55:09.859 139328 ERROR oslo_service.periodic_task >2018-06-25 05:55:39.731 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:55:39.732 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:55:39.739 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:55:39.740 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:55:39.772 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:55:39.772 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:55:39.772 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:55:39.772 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:55:39.772 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:55:39.772 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:55:39.772 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:55:39.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:55:39.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:55:39.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:55:49.776 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:55:49.808 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:55:49.808 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:55:49.808 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:55:49.808 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:55:49.808 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:55:49.808 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:55:49.808 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:55:49.810 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:55:49.811 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:55:49.811 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:56:09.811 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:56:09.846 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:56:09.846 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:56:09.846 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:56:09.846 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:56:09.846 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:56:09.846 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:56:09.846 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:56:09.848 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:56:09.849 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:56:09.849 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:56:09.849 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:56:09.849 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:56:09.849 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:56:09.849 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:56:09.849 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:56:09.849 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:56:09.849 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:56:09.849 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:56:09.849 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:56:09.849 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:56:09.849 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:56:09.849 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:56:09.849 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:56:09.849 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:56:09.849 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:56:09.849 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:56:09.849 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:56:09.849 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:56:09.849 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:56:09.849 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:56:09.849 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:56:09.849 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:56:09.849 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:56:09.849 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:56:09.849 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:56:09.849 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:56:09.849 139328 ERROR oslo_service.periodic_task >2018-06-25 05:56:39.731 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:56:39.732 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:56:39.739 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:56:39.739 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:56:39.775 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:56:39.775 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:56:39.775 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:56:39.775 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:56:39.775 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:56:39.775 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:56:39.775 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:56:39.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:56:39.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:56:39.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:56:49.779 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:56:49.812 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:56:49.812 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:56:49.812 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:56:49.812 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:56:49.812 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:56:49.812 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:56:49.812 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:56:49.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:56:49.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:56:49.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:57:09.817 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:57:09.849 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:57:09.849 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:57:09.849 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:57:09.849 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:57:09.849 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:57:09.849 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:57:09.849 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:57:09.851 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:57:09.851 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:57:09.852 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:57:09.852 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:57:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:57:09.852 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:57:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:57:09.852 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:57:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:57:09.852 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:57:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:57:09.852 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:57:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:57:09.852 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:57:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:57:09.852 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:57:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:57:09.852 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:57:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:57:09.852 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:57:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:57:09.852 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:57:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:57:09.852 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:57:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:57:09.852 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:57:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:57:09.852 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:57:09.852 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:57:09.852 139328 ERROR oslo_service.periodic_task >2018-06-25 05:57:39.733 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:57:39.734 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:57:39.740 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:57:39.741 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:57:39.774 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:57:39.774 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:57:39.774 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:57:39.774 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:57:39.774 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:57:39.774 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:57:39.774 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:57:39.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:57:39.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:57:39.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:57:49.777 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:57:49.811 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:57:49.811 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:57:49.811 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:57:49.811 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:57:49.811 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:57:49.811 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:57:49.811 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:57:49.813 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:57:49.814 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:57:49.814 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:58:09.815 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:58:09.848 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:58:09.848 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:58:09.848 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:58:09.848 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:58:09.848 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:58:09.848 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:58:09.848 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:58:09.850 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:58:09.850 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:58:09.851 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:58:09.851 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:58:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:58:09.851 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:58:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:58:09.851 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:58:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:58:09.851 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:58:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:58:09.851 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:58:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:58:09.851 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:58:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:58:09.851 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:58:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:58:09.851 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:58:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:58:09.851 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:58:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:58:09.851 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:58:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:58:09.851 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:58:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:58:09.851 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:58:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:58:09.851 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:58:09.851 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:58:09.851 139328 ERROR oslo_service.periodic_task >2018-06-25 05:58:39.732 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:58:39.733 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:58:39.741 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:58:39.742 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:58:39.776 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:58:39.776 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:58:39.776 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:58:39.776 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:58:39.776 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:58:39.776 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:58:39.776 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:58:39.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:58:39.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:58:39.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:58:49.780 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:58:49.815 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:58:49.815 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:58:49.815 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:58:49.815 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:58:49.815 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:58:49.815 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:58:49.815 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:58:49.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:58:49.818 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:58:49.818 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:59:09.819 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:59:09.855 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:59:09.855 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:59:09.855 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:59:09.855 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:59:09.855 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:59:09.855 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:59:09.855 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:59:09.858 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:59:09.858 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.116 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:59:09.858 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:59:09.858 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 05:59:09.858 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 05:59:09.858 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 05:59:09.858 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 05:59:09.858 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 05:59:09.858 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 05:59:09.858 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 05:59:09.858 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 05:59:09.858 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 05:59:09.858 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 05:59:09.858 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 05:59:09.858 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 05:59:09.858 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 05:59:09.858 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 05:59:09.858 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 05:59:09.858 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 05:59:09.858 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 05:59:09.858 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 05:59:09.858 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 05:59:09.858 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 05:59:09.858 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 05:59:09.858 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 05:59:09.858 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 05:59:09.858 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 05:59:09.858 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 05:59:09.858 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 05:59:09.858 139328 ERROR oslo_service.periodic_task >2018-06-25 05:59:39.730 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:59:39.730 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 05:59:39.738 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 05:59:39.738 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:59:39.770 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:59:39.770 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:59:39.770 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:59:39.770 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:59:39.770 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:59:39.770 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:59:39.770 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:59:39.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:59:39.772 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:59:39.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 05:59:49.773 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 05:59:49.815 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 05:59:49.815 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 05:59:49.815 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 05:59:49.815 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 05:59:49.815 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 05:59:49.815 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 05:59:49.815 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 05:59:49.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 05:59:49.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 05:59:49.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:00:09.818 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:00:09.853 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:00:09.853 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:00:09.853 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:00:09.853 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:00:09.853 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:00:09.853 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:00:09.853 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:00:09.856 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:00:09.856 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:00:09.856 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:00:09.856 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:00:09.856 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:00:09.856 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:00:09.856 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:00:09.856 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:00:09.856 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:00:09.856 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:00:09.856 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:00:09.856 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:00:09.856 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:00:09.856 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:00:09.856 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:00:09.856 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:00:09.856 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:00:09.856 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:00:09.856 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:00:09.856 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:00:09.856 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:00:09.856 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:00:09.856 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:00:09.856 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:00:09.856 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:00:09.856 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:00:09.856 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:00:09.856 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:00:09.856 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:00:09.856 139328 ERROR oslo_service.periodic_task >2018-06-25 06:00:39.728 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:00:39.729 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:00:39.737 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:00:39.738 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:00:39.775 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:00:39.775 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:00:39.775 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:00:39.775 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:00:39.775 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:00:39.775 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:00:39.775 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:00:39.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:00:39.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:00:39.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:00:49.778 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:00:49.815 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:00:49.815 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:00:49.815 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:00:49.815 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:00:49.815 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:00:49.815 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:00:49.815 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:00:49.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:00:49.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:00:49.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:01:09.817 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:01:09.850 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:01:09.850 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:01:09.850 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:01:09.850 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:01:09.850 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:01:09.850 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:01:09.850 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:01:09.852 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:01:09.852 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:01:09.852 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:01:09.852 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:01:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:01:09.852 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:01:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:01:09.852 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:01:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:01:09.852 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:01:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:01:09.852 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:01:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:01:09.852 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:01:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:01:09.852 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:01:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:01:09.852 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:01:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:01:09.852 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:01:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:01:09.852 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:01:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:01:09.852 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:01:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:01:09.852 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:01:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:01:09.852 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:01:09.852 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:01:09.852 139328 ERROR oslo_service.periodic_task >2018-06-25 06:01:39.724 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:01:39.725 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:01:39.732 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:01:39.732 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:01:39.777 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:01:39.777 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:01:39.777 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:01:39.777 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:01:39.777 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:01:39.777 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:01:39.777 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:01:39.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:01:39.779 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.046 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:01:39.780 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:01:49.780 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:01:49.814 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:01:49.814 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:01:49.814 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:01:49.814 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:01:49.814 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:01:49.814 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:01:49.814 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:01:49.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:01:49.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.084 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:01:49.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:02:09.817 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:02:09.850 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:02:09.850 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:02:09.850 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:02:09.850 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:02:09.850 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:02:09.850 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:02:09.850 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:02:09.852 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:02:09.852 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.119 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:02:09.852 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:02:09.852 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:02:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:02:09.852 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:02:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:02:09.852 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:02:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:02:09.852 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:02:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:02:09.852 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:02:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:02:09.852 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:02:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:02:09.852 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:02:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:02:09.852 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:02:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:02:09.852 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:02:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:02:09.852 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:02:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:02:09.852 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:02:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:02:09.852 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:02:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:02:09.852 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:02:09.852 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:02:09.852 139328 ERROR oslo_service.periodic_task >2018-06-25 06:02:39.724 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:02:39.725 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:02:39.742 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:02:39.742 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:02:39.794 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:02:39.794 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:02:39.794 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:02:39.794 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:02:39.794 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:02:39.794 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:02:39.794 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:02:39.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:02:39.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.053 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:02:39.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:02:49.797 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:02:49.831 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:02:49.831 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:02:49.831 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:02:49.831 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:02:49.831 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:02:49.831 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:02:49.831 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:02:49.833 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:02:49.834 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.091 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:02:49.834 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:03:09.835 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:03:09.866 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:03:09.866 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:03:09.866 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:03:09.866 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:03:09.866 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:03:09.866 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:03:09.866 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:03:09.868 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:03:09.869 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.126 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:03:09.869 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:03:09.869 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:03:09.869 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:03:09.869 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:03:09.869 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:03:09.869 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:03:09.869 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:03:09.869 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:03:09.869 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:03:09.869 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:03:09.869 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:03:09.869 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:03:09.869 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:03:09.869 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:03:09.869 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:03:09.869 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:03:09.869 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:03:09.869 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:03:09.869 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:03:09.869 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:03:09.869 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:03:09.869 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:03:09.869 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:03:09.869 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:03:09.869 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:03:09.869 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:03:09.869 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:03:09.869 139328 ERROR oslo_service.periodic_task >2018-06-25 06:03:39.721 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:03:39.721 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:03:39.728 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:03:39.729 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:03:39.763 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:03:39.763 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:03:39.763 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:03:39.763 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:03:39.763 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:03:39.763 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:03:39.763 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:03:39.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:03:39.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:03:39.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:03:49.766 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:03:49.800 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:03:49.800 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:03:49.800 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:03:49.800 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:03:49.800 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:03:49.800 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:03:49.800 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:03:49.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:03:49.803 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:03:49.803 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:04:09.803 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:04:09.835 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:04:09.835 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:04:09.835 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:04:09.835 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:04:09.835 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:04:09.835 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:04:09.835 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:04:09.837 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:04:09.837 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.107 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:04:09.837 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:04:09.837 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:04:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:04:09.837 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:04:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:04:09.837 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:04:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:04:09.837 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:04:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:04:09.837 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:04:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:04:09.837 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:04:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:04:09.837 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:04:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:04:09.837 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:04:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:04:09.837 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:04:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:04:09.837 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:04:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:04:09.837 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:04:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:04:09.837 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:04:09.837 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:04:09.837 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:04:09.837 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:04:09.837 139328 ERROR oslo_service.periodic_task >2018-06-25 06:04:39.719 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:04:39.719 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:04:39.727 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:04:39.728 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:04:39.765 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:04:39.765 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:04:39.765 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:04:39.765 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:04:39.765 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:04:39.765 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:04:39.765 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:04:39.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:04:39.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:04:39.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:04:49.769 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:04:49.802 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:04:49.802 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:04:49.802 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:04:49.802 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:04:49.802 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:04:49.802 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:04:49.802 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:04:49.804 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:04:49.804 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:04:49.804 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:05:09.805 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:05:09.837 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:05:09.837 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:05:09.837 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:05:09.837 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:05:09.837 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:05:09.837 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:05:09.837 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:05:09.839 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:05:09.839 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:05:09.840 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:05:09.840 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:05:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:05:09.840 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:05:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:05:09.840 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:05:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:05:09.840 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:05:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:05:09.840 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:05:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:05:09.840 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:05:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:05:09.840 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:05:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:05:09.840 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:05:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:05:09.840 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:05:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:05:09.840 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:05:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:05:09.840 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:05:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:05:09.840 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:05:09.840 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:05:09.840 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:05:09.840 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:05:09.840 139328 ERROR oslo_service.periodic_task >2018-06-25 06:05:39.721 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:05:39.722 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:05:39.730 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:05:39.730 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:05:39.766 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:05:39.766 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:05:39.766 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:05:39.766 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:05:39.766 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:05:39.766 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:05:39.766 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:05:39.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:05:39.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:05:39.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:05:49.770 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:05:49.805 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:05:49.805 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:05:49.805 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:05:49.805 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:05:49.805 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:05:49.805 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:05:49.805 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:05:49.807 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:05:49.808 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:05:49.808 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:06:09.808 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:06:09.842 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:06:09.842 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:06:09.842 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:06:09.842 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:06:09.842 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:06:09.842 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:06:09.842 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:06:09.843 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:06:09.844 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:06:09.844 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:06:09.844 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:06:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:06:09.844 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:06:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:06:09.844 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:06:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:06:09.844 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:06:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:06:09.844 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:06:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:06:09.844 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:06:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:06:09.844 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:06:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:06:09.844 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:06:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:06:09.844 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:06:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:06:09.844 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:06:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:06:09.844 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:06:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:06:09.844 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:06:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:06:09.844 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:06:09.844 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:06:09.844 139328 ERROR oslo_service.periodic_task >2018-06-25 06:06:39.726 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:06:39.727 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:06:39.739 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:06:39.739 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:06:39.773 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:06:39.773 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:06:39.773 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:06:39.773 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:06:39.773 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:06:39.773 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:06:39.773 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:06:39.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:06:39.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:06:39.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:06:49.778 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:06:49.813 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:06:49.813 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:06:49.813 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:06:49.813 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:06:49.813 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:06:49.813 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:06:49.813 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:06:49.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:06:49.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.076 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:06:49.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:07:09.817 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:07:09.848 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:07:09.848 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:07:09.848 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:07:09.848 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:07:09.848 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:07:09.848 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:07:09.848 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:07:09.850 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:07:09.850 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:07:09.850 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:07:09.850 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:07:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:07:09.850 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:07:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:07:09.850 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:07:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:07:09.850 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:07:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:07:09.850 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:07:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:07:09.850 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:07:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:07:09.850 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:07:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:07:09.850 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:07:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:07:09.850 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:07:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:07:09.850 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:07:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:07:09.850 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:07:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:07:09.850 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:07:09.850 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:07:09.850 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:07:09.850 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:07:09.850 139328 ERROR oslo_service.periodic_task >2018-06-25 06:07:39.722 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:07:39.723 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:07:39.729 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:07:39.730 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:07:39.762 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:07:39.762 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:07:39.762 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:07:39.762 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:07:39.762 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:07:39.762 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:07:39.762 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:07:39.764 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:07:39.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:07:39.765 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:07:49.765 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:07:49.800 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:07:49.800 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:07:49.800 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:07:49.800 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:07:49.800 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:07:49.800 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:07:49.800 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:07:49.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:07:49.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:07:49.803 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:08:09.803 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:08:09.841 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:08:09.841 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:08:09.841 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:08:09.841 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:08:09.841 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:08:09.841 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:08:09.841 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:08:09.843 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:08:09.843 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:08:09.844 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:08:09.844 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:08:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:08:09.844 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:08:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:08:09.844 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:08:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:08:09.844 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:08:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:08:09.844 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:08:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:08:09.844 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:08:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:08:09.844 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:08:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:08:09.844 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:08:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:08:09.844 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:08:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:08:09.844 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:08:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:08:09.844 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:08:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:08:09.844 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:08:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:08:09.844 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:08:09.844 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:08:09.844 139328 ERROR oslo_service.periodic_task >2018-06-25 06:08:39.725 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:08:39.726 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:08:39.733 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:08:39.734 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:08:39.782 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:08:39.782 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:08:39.782 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:08:39.782 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:08:39.782 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:08:39.782 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:08:39.782 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:08:39.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:08:39.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.051 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:08:39.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:08:49.785 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:08:49.819 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:08:49.819 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:08:49.819 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:08:49.819 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:08:49.819 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:08:49.819 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:08:49.819 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:08:49.821 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:08:49.821 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.087 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:08:49.821 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:09:09.822 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:09:09.859 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:09:09.859 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:09:09.859 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:09:09.859 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:09:09.859 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:09:09.859 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:09:09.859 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:09:09.861 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:09:09.862 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.128 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:09:09.862 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:09:09.862 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:09:09.862 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:09:09.862 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:09:09.862 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:09:09.862 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:09:09.862 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:09:09.862 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:09:09.862 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:09:09.862 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:09:09.862 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:09:09.862 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:09:09.862 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:09:09.862 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:09:09.862 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:09:09.862 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:09:09.862 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:09:09.862 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:09:09.862 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:09:09.862 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:09:09.862 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:09:09.862 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:09:09.862 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:09:09.862 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:09:09.862 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:09:09.862 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:09:09.862 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:09:09.862 139328 ERROR oslo_service.periodic_task >2018-06-25 06:09:39.723 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:09:39.724 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:09:39.731 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:09:39.732 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:09:39.764 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:09:39.764 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:09:39.764 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:09:39.764 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:09:39.764 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:09:39.764 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:09:39.764 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:09:39.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:09:39.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:09:39.767 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:09:49.768 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:09:49.803 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:09:49.803 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:09:49.803 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:09:49.803 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:09:49.803 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:09:49.803 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:09:49.803 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:09:49.805 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:09:49.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:09:49.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:10:09.806 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:10:09.840 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:10:09.840 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:10:09.840 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:10:09.840 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:10:09.840 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:10:09.840 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:10:09.840 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:10:09.842 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:10:09.842 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:10:09.843 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:10:09.843 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:10:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:10:09.843 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:10:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:10:09.843 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:10:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:10:09.843 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:10:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:10:09.843 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:10:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:10:09.843 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:10:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:10:09.843 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:10:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:10:09.843 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:10:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:10:09.843 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:10:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:10:09.843 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:10:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:10:09.843 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:10:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:10:09.843 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:10:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:10:09.843 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:10:09.843 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:10:09.843 139328 ERROR oslo_service.periodic_task >2018-06-25 06:10:39.724 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:10:39.724 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:10:39.732 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:10:39.732 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:10:39.772 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:10:39.772 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:10:39.772 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:10:39.772 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:10:39.772 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:10:39.772 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:10:39.772 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:10:39.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:10:39.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.042 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:10:39.776 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:10:49.776 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:10:49.812 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:10:49.812 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:10:49.812 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:10:49.812 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:10:49.812 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:10:49.812 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:10:49.812 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:10:49.813 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:10:49.814 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.081 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:10:49.814 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:11:09.814 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:11:09.849 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:11:09.849 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:11:09.849 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:11:09.849 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:11:09.849 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:11:09.849 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:11:09.849 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:11:09.851 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:11:09.851 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.119 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:11:09.852 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:11:09.852 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:11:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:11:09.852 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:11:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:11:09.852 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:11:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:11:09.852 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:11:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:11:09.852 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:11:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:11:09.852 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:11:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:11:09.852 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:11:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:11:09.852 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:11:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:11:09.852 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:11:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:11:09.852 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:11:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:11:09.852 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:11:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:11:09.852 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:11:09.852 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:11:09.852 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:11:09.852 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:11:09.852 139328 ERROR oslo_service.periodic_task >2018-06-25 06:11:39.723 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:11:39.724 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:11:39.730 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:11:39.730 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:11:39.766 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:11:39.766 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:11:39.766 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:11:39.766 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:11:39.766 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:11:39.766 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:11:39.766 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:11:39.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:11:39.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:11:39.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:11:49.769 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:11:49.803 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:11:49.803 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:11:49.803 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:11:49.803 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:11:49.803 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:11:49.803 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:11:49.803 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:11:49.805 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:11:49.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:11:49.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:12:09.806 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:12:09.839 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:12:09.839 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:12:09.839 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:12:09.839 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:12:09.839 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:12:09.839 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:12:09.839 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:12:09.841 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:12:09.842 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:12:09.842 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:12:09.842 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:12:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:12:09.842 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:12:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:12:09.842 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:12:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:12:09.842 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:12:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:12:09.842 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:12:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:12:09.842 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:12:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:12:09.842 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:12:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:12:09.842 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:12:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:12:09.842 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:12:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:12:09.842 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:12:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:12:09.842 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:12:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:12:09.842 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:12:09.842 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:12:09.842 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:12:09.842 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:12:09.842 139328 ERROR oslo_service.periodic_task >2018-06-25 06:12:39.723 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:12:39.724 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:12:39.731 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:12:39.731 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:12:39.764 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:12:39.764 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:12:39.764 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:12:39.764 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:12:39.764 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:12:39.764 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:12:39.764 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:12:39.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:12:39.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:12:39.766 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:12:49.767 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:12:49.802 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:12:49.802 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:12:49.802 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:12:49.802 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:12:49.802 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:12:49.802 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:12:49.802 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:12:49.804 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:12:49.805 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:12:49.805 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:13:09.806 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:13:09.840 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:13:09.840 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:13:09.840 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:13:09.840 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:13:09.840 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:13:09.840 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:13:09.840 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:13:09.842 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:13:09.843 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:13:09.843 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:13:09.843 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:13:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:13:09.843 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:13:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:13:09.843 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:13:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:13:09.843 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:13:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:13:09.843 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:13:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:13:09.843 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:13:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:13:09.843 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:13:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:13:09.843 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:13:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:13:09.843 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:13:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:13:09.843 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:13:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:13:09.843 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:13:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:13:09.843 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:13:09.843 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:13:09.843 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:13:09.843 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:13:09.843 139328 ERROR oslo_service.periodic_task >2018-06-25 06:13:39.724 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:13:39.725 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:13:39.731 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:13:39.732 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:13:39.766 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:13:39.766 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:13:39.766 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:13:39.766 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:13:39.766 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:13:39.766 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:13:39.766 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:13:39.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:13:39.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:13:39.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:13:49.769 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:13:49.802 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:13:49.802 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:13:49.802 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:13:49.802 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:13:49.802 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:13:49.802 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:13:49.802 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:13:49.804 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:13:49.804 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:13:49.805 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:14:09.805 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:14:09.840 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:14:09.840 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:14:09.840 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:14:09.840 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:14:09.840 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:14:09.840 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:14:09.840 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:14:09.843 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:14:09.843 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:14:09.844 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:14:09.844 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:14:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:14:09.844 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:14:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:14:09.844 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:14:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:14:09.844 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:14:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:14:09.844 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:14:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:14:09.844 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:14:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:14:09.844 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:14:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:14:09.844 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:14:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:14:09.844 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:14:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:14:09.844 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:14:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:14:09.844 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:14:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:14:09.844 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:14:09.844 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:14:09.844 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:14:09.844 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:14:09.844 139328 ERROR oslo_service.periodic_task >2018-06-25 06:14:39.726 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:14:39.726 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:14:39.733 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:14:39.734 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:14:39.766 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:14:39.766 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:14:39.766 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:14:39.766 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:14:39.766 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:14:39.766 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:14:39.766 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:14:39.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:14:39.768 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:14:39.769 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:14:49.769 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:14:49.805 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:14:49.805 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:14:49.805 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:14:49.805 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:14:49.805 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:14:49.805 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:14:49.805 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:14:49.807 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:14:49.808 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:14:49.808 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:15:09.809 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:15:09.843 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:15:09.843 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:15:09.843 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:15:09.843 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:15:09.843 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:15:09.843 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:15:09.843 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:15:09.845 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:15:09.846 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:15:09.846 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:15:09.846 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:15:09.846 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:15:09.846 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:15:09.846 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:15:09.846 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:15:09.846 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:15:09.846 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:15:09.846 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:15:09.846 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:15:09.846 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:15:09.846 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:15:09.846 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:15:09.846 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:15:09.846 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:15:09.846 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:15:09.846 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:15:09.846 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:15:09.846 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:15:09.846 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:15:09.846 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:15:09.846 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:15:09.846 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:15:09.846 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:15:09.846 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:15:09.846 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:15:09.846 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:15:09.846 139328 ERROR oslo_service.periodic_task >2018-06-25 06:15:39.727 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:15:39.728 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:15:39.741 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:15:39.741 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:15:39.775 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:15:39.775 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:15:39.775 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:15:39.775 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:15:39.775 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:15:39.775 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:15:39.775 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:15:39.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:15:39.777 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:15:39.778 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:15:49.778 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:15:49.811 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:15:49.811 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:15:49.811 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:15:49.811 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:15:49.811 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:15:49.811 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:15:49.811 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:15:49.813 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:15:49.814 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:15:49.814 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:16:09.814 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:16:09.848 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:16:09.848 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:16:09.848 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:16:09.848 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:16:09.848 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:16:09.848 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:16:09.848 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:16:09.850 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:16:09.850 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:16:09.851 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:16:09.851 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:16:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:16:09.851 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:16:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:16:09.851 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:16:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:16:09.851 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:16:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:16:09.851 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:16:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:16:09.851 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:16:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:16:09.851 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:16:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:16:09.851 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:16:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:16:09.851 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:16:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:16:09.851 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:16:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:16:09.851 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:16:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:16:09.851 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:16:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:16:09.851 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:16:09.851 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:16:09.851 139328 ERROR oslo_service.periodic_task >2018-06-25 06:16:39.732 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:16:39.733 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:16:39.740 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:16:39.740 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:16:39.773 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:16:39.773 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:16:39.773 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:16:39.773 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:16:39.773 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:16:39.773 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:16:39.773 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:16:39.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:16:39.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:16:39.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:16:49.776 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:16:49.819 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:16:49.819 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:16:49.819 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:16:49.819 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:16:49.819 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:16:49.819 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:16:49.819 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:16:49.822 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:16:49.822 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.081 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:16:49.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:17:09.823 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:17:09.858 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:17:09.858 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:17:09.858 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:17:09.858 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:17:09.858 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:17:09.858 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:17:09.858 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:17:09.860 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:17:09.861 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.12 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:17:09.861 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:17:09.861 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:17:09.861 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:17:09.861 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:17:09.861 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:17:09.861 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:17:09.861 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:17:09.861 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:17:09.861 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:17:09.861 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:17:09.861 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:17:09.861 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:17:09.861 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:17:09.861 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:17:09.861 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:17:09.861 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:17:09.861 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:17:09.861 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:17:09.861 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:17:09.861 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:17:09.861 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:17:09.861 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:17:09.861 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:17:09.861 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:17:09.861 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:17:09.861 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:17:09.861 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:17:09.861 139328 ERROR oslo_service.periodic_task >2018-06-25 06:17:39.732 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:17:39.733 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:17:39.739 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:17:39.740 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:17:39.771 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:17:39.771 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:17:39.771 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:17:39.771 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:17:39.771 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:17:39.771 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:17:39.771 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:17:39.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:17:39.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.033 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:17:39.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:17:49.774 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:17:49.820 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:17:49.820 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:17:49.820 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:17:49.820 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:17:49.820 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:17:49.820 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:17:49.820 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:17:49.822 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:17:49.822 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.082 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:17:49.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:18:09.823 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:18:09.856 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:18:09.856 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:18:09.856 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:18:09.856 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:18:09.856 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:18:09.856 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:18:09.856 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:18:09.858 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:18:09.858 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.119 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:18:09.859 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:18:09.859 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:18:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:18:09.859 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:18:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:18:09.859 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:18:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:18:09.859 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:18:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:18:09.859 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:18:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:18:09.859 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:18:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:18:09.859 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:18:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:18:09.859 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:18:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:18:09.859 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:18:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:18:09.859 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:18:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:18:09.859 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:18:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:18:09.859 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:18:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:18:09.859 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:18:09.859 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:18:09.859 139328 ERROR oslo_service.periodic_task >2018-06-25 06:18:39.730 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:18:39.731 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:18:39.737 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:18:39.737 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:18:39.773 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:18:39.773 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:18:39.773 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:18:39.773 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:18:39.773 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:18:39.773 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:18:39.773 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:18:39.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:18:39.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:18:39.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:18:49.776 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:18:49.816 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:18:49.816 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:18:49.816 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:18:49.816 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:18:49.816 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:18:49.816 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:18:49.816 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:18:49.818 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:18:49.819 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.082 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:18:49.819 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:19:09.820 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:19:09.854 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:19:09.854 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:19:09.854 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:19:09.854 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:19:09.854 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:19:09.854 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:19:09.854 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:19:09.856 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:19:09.856 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.119 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:19:09.856 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:19:09.856 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:19:09.856 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:19:09.856 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:19:09.856 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:19:09.856 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:19:09.856 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:19:09.856 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:19:09.856 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:19:09.856 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:19:09.856 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:19:09.856 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:19:09.856 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:19:09.856 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:19:09.856 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:19:09.856 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:19:09.856 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:19:09.856 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:19:09.856 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:19:09.856 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:19:09.856 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:19:09.856 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:19:09.856 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:19:09.856 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:19:09.856 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:19:09.856 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:19:09.856 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:19:09.856 139328 ERROR oslo_service.periodic_task >2018-06-25 06:19:39.728 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:19:39.728 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:19:39.735 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:19:39.736 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:19:39.771 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:19:39.771 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:19:39.771 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:19:39.771 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:19:39.771 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:19:39.771 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:19:39.771 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:19:39.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:19:39.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:19:39.774 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:19:49.774 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:19:49.809 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:19:49.809 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:19:49.809 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:19:49.809 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:19:49.809 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:19:49.809 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:19:49.809 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:19:49.811 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:19:49.811 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:19:49.812 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:20:09.812 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:20:09.846 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:20:09.846 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:20:09.846 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:20:09.846 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:20:09.846 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:20:09.846 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:20:09.846 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:20:09.848 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:20:09.849 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:20:09.849 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:20:09.849 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:20:09.849 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:20:09.849 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:20:09.849 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:20:09.849 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:20:09.849 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:20:09.849 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:20:09.849 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:20:09.849 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:20:09.849 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:20:09.849 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:20:09.849 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:20:09.849 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:20:09.849 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:20:09.849 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:20:09.849 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:20:09.849 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:20:09.849 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:20:09.849 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:20:09.849 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:20:09.849 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:20:09.849 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:20:09.849 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:20:09.849 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:20:09.849 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:20:09.849 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:20:09.849 139328 ERROR oslo_service.periodic_task >2018-06-25 06:20:39.730 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:20:39.731 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:20:39.737 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:20:39.738 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:20:39.771 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:20:39.771 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:20:39.771 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:20:39.771 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:20:39.771 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:20:39.771 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:20:39.771 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:20:39.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:20:39.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:20:39.773 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:20:49.774 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:20:49.808 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:20:49.808 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:20:49.808 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:20:49.808 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:20:49.808 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:20:49.808 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:20:49.808 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:20:49.811 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:20:49.811 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:20:49.811 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:21:09.812 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:21:09.845 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:21:09.845 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:21:09.845 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:21:09.845 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:21:09.845 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:21:09.845 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:21:09.845 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:21:09.847 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:21:09.847 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:21:09.848 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:21:09.848 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:21:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:21:09.848 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:21:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:21:09.848 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:21:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:21:09.848 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:21:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:21:09.848 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:21:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:21:09.848 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:21:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:21:09.848 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:21:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:21:09.848 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:21:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:21:09.848 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:21:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:21:09.848 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:21:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:21:09.848 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:21:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:21:09.848 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:21:09.848 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:21:09.848 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:21:09.848 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:21:09.848 139328 ERROR oslo_service.periodic_task >2018-06-25 06:21:39.729 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:21:39.730 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:21:39.739 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:21:39.739 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:21:39.772 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:21:39.772 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:21:39.772 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:21:39.772 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:21:39.772 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:21:39.772 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:21:39.772 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:21:39.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:21:39.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:21:39.775 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:21:49.776 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:21:49.810 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:21:49.810 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:21:49.810 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:21:49.810 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:21:49.810 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:21:49.810 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:21:49.810 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:21:49.813 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:21:49.813 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:21:49.814 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:22:09.814 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:22:09.848 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:22:09.848 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:22:09.848 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:22:09.848 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:22:09.848 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:22:09.848 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:22:09.848 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:22:09.850 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:22:09.851 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:22:09.851 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:22:09.851 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:22:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:22:09.851 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:22:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:22:09.851 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:22:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:22:09.851 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:22:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:22:09.851 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:22:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:22:09.851 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:22:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:22:09.851 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:22:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:22:09.851 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:22:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:22:09.851 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:22:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:22:09.851 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:22:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:22:09.851 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:22:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:22:09.851 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:22:09.851 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:22:09.851 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:22:09.851 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:22:09.851 139328 ERROR oslo_service.periodic_task >2018-06-25 06:22:39.733 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:22:39.734 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:22:39.742 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:22:39.743 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:22:39.777 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:22:39.777 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:22:39.777 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:22:39.777 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:22:39.777 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:22:39.777 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:22:39.777 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:22:39.781 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:22:39.781 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:22:39.781 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:22:49.782 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:22:49.814 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:22:49.814 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:22:49.814 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:22:49.814 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:22:49.814 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:22:49.814 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:22:49.814 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:22:49.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:22:49.816 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:22:49.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:23:09.817 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:23:09.851 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:23:09.851 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:23:09.851 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:23:09.851 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:23:09.851 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:23:09.851 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:23:09.851 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:23:09.853 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:23:09.853 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:23:09.853 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:23:09.853 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:23:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:23:09.853 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:23:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:23:09.853 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:23:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:23:09.853 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:23:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:23:09.853 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:23:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:23:09.853 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:23:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:23:09.853 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:23:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:23:09.853 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:23:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:23:09.853 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:23:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:23:09.853 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:23:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:23:09.853 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:23:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:23:09.853 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:23:09.853 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:23:09.853 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:23:09.853 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:23:09.853 139328 ERROR oslo_service.periodic_task >2018-06-25 06:23:39.735 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:23:39.736 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:23:39.746 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:23:39.746 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:23:39.785 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:23:39.785 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:23:39.785 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:23:39.785 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:23:39.785 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:23:39.785 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:23:39.785 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:23:39.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:23:39.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:23:39.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:23:49.788 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:23:49.824 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:23:49.824 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:23:49.824 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:23:49.824 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:23:49.824 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:23:49.824 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:23:49.824 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:23:49.827 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:23:49.827 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.08 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:23:49.827 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:24:09.828 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:24:09.873 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:24:09.873 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:24:09.873 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:24:09.873 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:24:09.873 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:24:09.873 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:24:09.873 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:24:09.876 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:24:09.876 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.129 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:24:09.877 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:24:09.877 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:24:09.877 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:24:09.877 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:24:09.877 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:24:09.877 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:24:09.877 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:24:09.877 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:24:09.877 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:24:09.877 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:24:09.877 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:24:09.877 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:24:09.877 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:24:09.877 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:24:09.877 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:24:09.877 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:24:09.877 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:24:09.877 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:24:09.877 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:24:09.877 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:24:09.877 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:24:09.877 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:24:09.877 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:24:09.877 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:24:09.877 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:24:09.877 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:24:09.877 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:24:09.877 139328 ERROR oslo_service.periodic_task >2018-06-25 06:24:39.738 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:24:39.739 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:24:39.748 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:24:39.749 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:24:39.792 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:24:39.792 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:24:39.792 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:24:39.792 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:24:39.792 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:24:39.792 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:24:39.792 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:24:39.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:24:39.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.046 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:24:39.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:24:49.795 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:24:49.827 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:24:49.827 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:24:49.827 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:24:49.827 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:24:49.827 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:24:49.827 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:24:49.827 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:24:49.829 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:24:49.830 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.081 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:24:49.830 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:25:09.831 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:25:09.879 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:25:09.879 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:25:09.879 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:25:09.879 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:25:09.879 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:25:09.879 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:25:09.879 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:25:09.883 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:25:09.883 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.134 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:25:09.884 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:25:09.884 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:25:09.884 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:25:09.884 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:25:09.884 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:25:09.884 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:25:09.884 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:25:09.884 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:25:09.884 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:25:09.884 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:25:09.884 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:25:09.884 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:25:09.884 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:25:09.884 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:25:09.884 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:25:09.884 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:25:09.884 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:25:09.884 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:25:09.884 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:25:09.884 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:25:09.884 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:25:09.884 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:25:09.884 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:25:09.884 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:25:09.884 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:25:09.884 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:25:09.884 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:25:09.884 139328 ERROR oslo_service.periodic_task >2018-06-25 06:25:39.735 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:25:39.736 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:25:39.743 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:25:39.744 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:25:39.782 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:25:39.782 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:25:39.782 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:25:39.782 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:25:39.782 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:25:39.782 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:25:39.782 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:25:39.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:25:39.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.041 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:25:39.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:25:49.786 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:25:49.826 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:25:49.826 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:25:49.826 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:25:49.826 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:25:49.826 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:25:49.826 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:25:49.826 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:25:49.828 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:25:49.829 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.085 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:25:49.829 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:26:09.829 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:26:09.865 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:26:09.865 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:26:09.865 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:26:09.865 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:26:09.865 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:26:09.865 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:26:09.865 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:26:09.867 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:26:09.867 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.124 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:26:09.868 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:26:09.868 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:26:09.868 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:26:09.868 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:26:09.868 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:26:09.868 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:26:09.868 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:26:09.868 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:26:09.868 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:26:09.868 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:26:09.868 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:26:09.868 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:26:09.868 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:26:09.868 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:26:09.868 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:26:09.868 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:26:09.868 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:26:09.868 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:26:09.868 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:26:09.868 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:26:09.868 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:26:09.868 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:26:09.868 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:26:09.868 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:26:09.868 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:26:09.868 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:26:09.868 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:26:09.868 139328 ERROR oslo_service.periodic_task >2018-06-25 06:26:39.739 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:26:39.740 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:26:39.747 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:26:39.748 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:26:39.782 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:26:39.782 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:26:39.782 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:26:39.782 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:26:39.782 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:26:39.782 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:26:39.782 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:26:39.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:26:39.784 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:26:39.785 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:26:49.785 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:26:49.820 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:26:49.820 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:26:49.820 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:26:49.820 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:26:49.820 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:26:49.820 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:26:49.820 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:26:49.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:26:49.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:26:49.824 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:27:09.824 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:27:09.857 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:27:09.857 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:27:09.857 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:27:09.857 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:27:09.857 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:27:09.857 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:27:09.857 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:27:09.860 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:27:09.860 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:27:09.860 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:27:09.860 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:27:09.860 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:27:09.860 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:27:09.860 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:27:09.860 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:27:09.860 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:27:09.860 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:27:09.860 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:27:09.860 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:27:09.860 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:27:09.860 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:27:09.860 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:27:09.860 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:27:09.860 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:27:09.860 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:27:09.860 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:27:09.860 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:27:09.860 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:27:09.860 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:27:09.860 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:27:09.860 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:27:09.860 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:27:09.860 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:27:09.860 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:27:09.860 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:27:09.860 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:27:09.860 139328 ERROR oslo_service.periodic_task >2018-06-25 06:27:39.742 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:27:39.742 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:27:39.749 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:27:39.750 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:27:39.785 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:27:39.785 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:27:39.785 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:27:39.785 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:27:39.785 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:27:39.785 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:27:39.785 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:27:39.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:27:39.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:27:39.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:27:49.788 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:27:49.819 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:27:49.819 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:27:49.819 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:27:49.819 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:27:49.819 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:27:49.819 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:27:49.819 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:27:49.821 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:27:49.822 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:27:49.822 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:28:09.822 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:28:09.856 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:28:09.856 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:28:09.856 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:28:09.856 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:28:09.856 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:28:09.856 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:28:09.856 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:28:09.858 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:28:09.858 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.108 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:28:09.858 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:28:09.858 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:28:09.858 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:28:09.858 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:28:09.858 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:28:09.858 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:28:09.858 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:28:09.858 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:28:09.858 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:28:09.858 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:28:09.858 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:28:09.858 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:28:09.858 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:28:09.858 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:28:09.858 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:28:09.858 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:28:09.858 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:28:09.858 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:28:09.858 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:28:09.858 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:28:09.858 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:28:09.858 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:28:09.858 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:28:09.858 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:28:09.858 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:28:09.858 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:28:09.858 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:28:09.858 139328 ERROR oslo_service.periodic_task >2018-06-25 06:28:39.740 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:28:39.740 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:28:39.747 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:28:39.748 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:28:39.780 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:28:39.780 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:28:39.780 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:28:39.780 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:28:39.780 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:28:39.780 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:28:39.780 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:28:39.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:28:39.782 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:28:39.783 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:28:49.783 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:28:49.820 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:28:49.820 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:28:49.820 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:28:49.820 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:28:49.820 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:28:49.820 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:28:49.820 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:28:49.822 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:28:49.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:28:49.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:29:09.824 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:29:09.857 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:29:09.857 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:29:09.857 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:29:09.857 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:29:09.857 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:29:09.857 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:29:09.857 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:29:09.859 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:29:09.859 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:29:09.859 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:29:09.859 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:29:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:29:09.859 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:29:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:29:09.859 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:29:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:29:09.859 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:29:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:29:09.859 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:29:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:29:09.859 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:29:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:29:09.859 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:29:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:29:09.859 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:29:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:29:09.859 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:29:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:29:09.859 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:29:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:29:09.859 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:29:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:29:09.859 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:29:09.859 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:29:09.859 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:29:09.859 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:29:09.859 139328 ERROR oslo_service.periodic_task >2018-06-25 06:29:39.741 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:29:39.742 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:29:39.748 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:29:39.749 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:29:39.795 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:29:39.795 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:29:39.795 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:29:39.795 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:29:39.795 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:29:39.795 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:29:39.795 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:29:39.797 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:29:39.798 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.049 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:29:39.798 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:29:49.799 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:29:49.833 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:29:49.833 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:29:49.833 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:29:49.833 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:29:49.833 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:29:49.833 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:29:49.833 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:29:49.836 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:29:49.836 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.087 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:29:49.836 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:30:09.837 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:30:09.877 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:30:09.877 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:30:09.877 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:30:09.877 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:30:09.877 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:30:09.877 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:30:09.877 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:30:09.879 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:30:09.879 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.131 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:30:09.880 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:30:09.880 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:30:09.880 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:30:09.880 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:30:09.880 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:30:09.880 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:30:09.880 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:30:09.880 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:30:09.880 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:30:09.880 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:30:09.880 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:30:09.880 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:30:09.880 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:30:09.880 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:30:09.880 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:30:09.880 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:30:09.880 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:30:09.880 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:30:09.880 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:30:09.880 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:30:09.880 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:30:09.880 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:30:09.880 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:30:09.880 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:30:09.880 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:30:09.880 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:30:09.880 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:30:09.880 139328 ERROR oslo_service.periodic_task >2018-06-25 06:30:39.741 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:30:39.743 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:30:39.749 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:30:39.750 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:30:39.784 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:30:39.784 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:30:39.784 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:30:39.784 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:30:39.784 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:30:39.784 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:30:39.784 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:30:39.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:30:39.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:30:39.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:30:49.787 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:30:49.824 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:30:49.824 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:30:49.824 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:30:49.824 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:30:49.824 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:30:49.824 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:30:49.824 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:30:49.827 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:30:49.827 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:30:49.828 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:31:09.828 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:31:09.859 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:31:09.859 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:31:09.859 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:31:09.859 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:31:09.859 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:31:09.859 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:31:09.859 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:31:09.862 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:31:09.862 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:31:09.863 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:31:09.863 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:31:09.863 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:31:09.863 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:31:09.863 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:31:09.863 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:31:09.863 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:31:09.863 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:31:09.863 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:31:09.863 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:31:09.863 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:31:09.863 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:31:09.863 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:31:09.863 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:31:09.863 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:31:09.863 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:31:09.863 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:31:09.863 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:31:09.863 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:31:09.863 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:31:09.863 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:31:09.863 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:31:09.863 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:31:09.863 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:31:09.863 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:31:09.863 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:31:09.863 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:31:09.863 139328 ERROR oslo_service.periodic_task >2018-06-25 06:31:39.744 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:31:39.745 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:31:39.752 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:31:39.752 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:31:39.785 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:31:39.785 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:31:39.785 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:31:39.785 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:31:39.785 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:31:39.785 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:31:39.785 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:31:39.786 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:31:39.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:31:39.787 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:31:49.787 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:31:49.821 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:31:49.821 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:31:49.821 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:31:49.821 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:31:49.821 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:31:49.821 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:31:49.821 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:31:49.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:31:49.824 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:31:49.824 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:32:09.824 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:32:09.859 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:32:09.859 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:32:09.859 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:32:09.859 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:32:09.859 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:32:09.859 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:32:09.859 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:32:09.862 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:32:09.862 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:32:09.862 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:32:09.862 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:32:09.862 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:32:09.862 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:32:09.862 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:32:09.862 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:32:09.862 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:32:09.862 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:32:09.862 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:32:09.862 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:32:09.862 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:32:09.862 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:32:09.862 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:32:09.862 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:32:09.862 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:32:09.862 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:32:09.862 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:32:09.862 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:32:09.862 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:32:09.862 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:32:09.862 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:32:09.862 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:32:09.862 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:32:09.862 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:32:09.862 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:32:09.862 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:32:09.862 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:32:09.862 139328 ERROR oslo_service.periodic_task >2018-06-25 06:32:39.744 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:32:39.745 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:32:39.753 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:32:39.753 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:32:39.786 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:32:39.786 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:32:39.786 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:32:39.786 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:32:39.786 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:32:39.786 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:32:39.786 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:32:39.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:32:39.788 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:32:39.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:32:49.789 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:32:49.822 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:32:49.822 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:32:49.822 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:32:49.822 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:32:49.822 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:32:49.822 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:32:49.822 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:32:49.825 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:32:49.825 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.071 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:32:49.826 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:33:09.826 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:33:09.862 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:33:09.862 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:33:09.862 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:33:09.862 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:33:09.862 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:33:09.862 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:33:09.862 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:33:09.865 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:33:09.865 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:33:09.865 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:33:09.865 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:33:09.865 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:33:09.865 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:33:09.865 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:33:09.865 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:33:09.865 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:33:09.865 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:33:09.865 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:33:09.865 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:33:09.865 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:33:09.865 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:33:09.865 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:33:09.865 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:33:09.865 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:33:09.865 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:33:09.865 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:33:09.865 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:33:09.865 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:33:09.865 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:33:09.865 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:33:09.865 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:33:09.865 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:33:09.865 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:33:09.865 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:33:09.865 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:33:09.865 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:33:09.865 139328 ERROR oslo_service.periodic_task >2018-06-25 06:33:39.747 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:33:39.747 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:33:39.754 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:33:39.755 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:33:39.787 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:33:39.787 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:33:39.787 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:33:39.787 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:33:39.787 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:33:39.787 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:33:39.787 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:33:39.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:33:39.789 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:33:39.790 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:33:49.790 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:33:49.827 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:33:49.827 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:33:49.827 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:33:49.827 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:33:49.827 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:33:49.827 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:33:49.827 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:33:49.829 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:33:49.830 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:33:49.830 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:34:09.831 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:34:09.863 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:34:09.863 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:34:09.863 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:34:09.863 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:34:09.863 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:34:09.863 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:34:09.863 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:34:09.864 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:34:09.865 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:34:09.865 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:34:09.865 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:34:09.865 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:34:09.865 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:34:09.865 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:34:09.865 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:34:09.865 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:34:09.865 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:34:09.865 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:34:09.865 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:34:09.865 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:34:09.865 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:34:09.865 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:34:09.865 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:34:09.865 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:34:09.865 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:34:09.865 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:34:09.865 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:34:09.865 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:34:09.865 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:34:09.865 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:34:09.865 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:34:09.865 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:34:09.865 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:34:09.865 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:34:09.865 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:34:09.865 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:34:09.865 139328 ERROR oslo_service.periodic_task >2018-06-25 06:34:39.746 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:34:39.747 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:34:39.753 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:34:39.754 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:34:39.790 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:34:39.790 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:34:39.790 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:34:39.790 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:34:39.790 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:34:39.790 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:34:39.790 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:34:39.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:34:39.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:34:39.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:34:49.794 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:34:49.843 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:34:49.843 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:34:49.843 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:34:49.843 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:34:49.843 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:34:49.843 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:34:49.843 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:34:49.845 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:34:49.846 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.092 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:34:49.846 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:35:09.847 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:35:09.881 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:35:09.881 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:35:09.881 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:35:09.881 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:35:09.881 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:35:09.881 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:35:09.881 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:35:09.884 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:35:09.885 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.13 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:35:09.885 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:35:09.885 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:35:09.885 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:35:09.885 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:35:09.885 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:35:09.885 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:35:09.885 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:35:09.885 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:35:09.885 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:35:09.885 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:35:09.885 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:35:09.885 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:35:09.885 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:35:09.885 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:35:09.885 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:35:09.885 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:35:09.885 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:35:09.885 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:35:09.885 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:35:09.885 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:35:09.885 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:35:09.885 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:35:09.885 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:35:09.885 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:35:09.885 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:35:09.885 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:35:09.885 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:35:09.885 139328 ERROR oslo_service.periodic_task >2018-06-25 06:35:39.747 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:35:39.747 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:35:39.760 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:35:39.761 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:35:39.799 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:35:39.799 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:35:39.799 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:35:39.799 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:35:39.799 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:35:39.799 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:35:39.799 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:35:39.801 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:35:39.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.04 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:35:39.802 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:35:49.803 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:35:49.839 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:35:49.839 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:35:49.839 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:35:49.839 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:35:49.839 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:35:49.839 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:35:49.839 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:35:49.841 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:35:49.841 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.08 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:35:49.842 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:36:09.842 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:36:09.877 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:36:09.877 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:36:09.877 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:36:09.877 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:36:09.877 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:36:09.877 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:36:09.877 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:36:09.879 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:36:09.879 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.117 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:36:09.880 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:36:09.880 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:36:09.880 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:36:09.880 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:36:09.880 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:36:09.880 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:36:09.880 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:36:09.880 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:36:09.880 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:36:09.880 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:36:09.880 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:36:09.880 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:36:09.880 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:36:09.880 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:36:09.880 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:36:09.880 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:36:09.880 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:36:09.880 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:36:09.880 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:36:09.880 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:36:09.880 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:36:09.880 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:36:09.880 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:36:09.880 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:36:09.880 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:36:09.880 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:36:09.880 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:36:09.880 139328 ERROR oslo_service.periodic_task >2018-06-25 06:36:39.751 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:36:39.752 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:37:39.749 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:37:39.750 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:37:39.757 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:37:39.758 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:37:39.793 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:37:39.793 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:37:39.793 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:37:39.793 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:37:39.793 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:37:39.793 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:37:39.793 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:37:39.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:37:39.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:37:39.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:37:49.797 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:37:49.849 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:37:49.849 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:37:49.849 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:37:49.849 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:37:49.849 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:37:49.849 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:37:49.849 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:37:49.851 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:37:49.852 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.094 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:37:49.852 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:38:09.852 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:38:09.885 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:38:09.885 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:38:09.885 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:38:09.885 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:38:09.885 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:38:09.885 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:38:09.885 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:38:09.887 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:38:09.887 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.129 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:38:09.887 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:38:09.887 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:38:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:38:09.887 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:38:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:38:09.887 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:38:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:38:09.887 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:38:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:38:09.887 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:38:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:38:09.887 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:38:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:38:09.887 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:38:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:38:09.887 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:38:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:38:09.887 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:38:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:38:09.887 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:38:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:38:09.887 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:38:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:38:09.887 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:38:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:38:09.887 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:38:09.887 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:38:09.887 139328 ERROR oslo_service.periodic_task >2018-06-25 06:38:39.749 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:38:39.750 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:38:39.757 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:38:39.757 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:38:39.791 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:38:39.791 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:38:39.791 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:38:39.791 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:38:39.791 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:38:39.791 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:38:39.791 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:38:39.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:38:39.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:38:39.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:38:49.794 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:38:49.834 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:38:49.834 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:38:49.834 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:38:49.834 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:38:49.834 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:38:49.834 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:38:49.834 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:38:49.836 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:38:49.837 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:38:49.837 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:39:09.838 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:39:09.871 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:39:09.871 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:39:09.871 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:39:09.871 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:39:09.871 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:39:09.871 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:39:09.871 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:39:09.872 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:39:09.872 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:39:09.873 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:39:09.873 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:39:09.873 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:39:09.873 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:39:09.873 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:39:09.873 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:39:09.873 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:39:09.873 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:39:09.873 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:39:09.873 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:39:09.873 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:39:09.873 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:39:09.873 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:39:09.873 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:39:09.873 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:39:09.873 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:39:09.873 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:39:09.873 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:39:09.873 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:39:09.873 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:39:09.873 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:39:09.873 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:39:09.873 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:39:09.873 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:39:09.873 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:39:09.873 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:39:09.873 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:39:09.873 139328 ERROR oslo_service.periodic_task >2018-06-25 06:39:39.744 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:39:39.745 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:39:39.753 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:39:39.754 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:39:39.788 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:39:39.788 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:39:39.788 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:39:39.788 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:39:39.788 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:39:39.788 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:39:39.788 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:39:39.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:39:39.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:39:39.791 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:39:49.792 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:39:49.826 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:39:49.826 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:39:49.826 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:39:49.826 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:39:49.826 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:39:49.826 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:39:49.826 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:39:49.828 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:39:49.828 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:39:49.829 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:40:09.829 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:40:09.864 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:40:09.864 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:40:09.864 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:40:09.864 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:40:09.864 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:40:09.864 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:40:09.864 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:40:09.866 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:40:09.866 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:40:09.866 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:40:09.866 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:40:09.866 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:40:09.866 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:40:09.866 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:40:09.866 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:40:09.866 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:40:09.866 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:40:09.866 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:40:09.866 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:40:09.866 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:40:09.866 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:40:09.866 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:40:09.866 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:40:09.866 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:40:09.866 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:40:09.866 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:40:09.866 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:40:09.866 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:40:09.866 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:40:09.866 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:40:09.866 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:40:09.866 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:40:09.866 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:40:09.866 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:40:09.866 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:40:09.866 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:40:09.866 139328 ERROR oslo_service.periodic_task >2018-06-25 06:40:39.748 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:40:39.748 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:40:39.760 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:40:39.760 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:40:39.792 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:40:39.792 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:40:39.792 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:40:39.792 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:40:39.792 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:40:39.792 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:40:39.792 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:40:39.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:40:39.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.033 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:40:39.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:40:49.795 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:40:49.827 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:40:49.827 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:40:49.827 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:40:49.827 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:40:49.827 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:40:49.827 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:40:49.827 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:40:49.829 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:40:49.830 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.069 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:40:49.830 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:41:09.830 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:41:09.865 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:41:09.865 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:41:09.865 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:41:09.865 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:41:09.865 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:41:09.865 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:41:09.865 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:41:09.867 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:41:09.867 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.106 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:41:09.867 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:41:09.867 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:41:09.867 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:41:09.867 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:41:09.867 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:41:09.867 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:41:09.867 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:41:09.867 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:41:09.867 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:41:09.867 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:41:09.867 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:41:09.867 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:41:09.867 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:41:09.867 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:41:09.867 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:41:09.867 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:41:09.867 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:41:09.867 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:41:09.867 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:41:09.867 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:41:09.867 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:41:09.867 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:41:09.867 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:41:09.867 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:41:09.867 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:41:09.867 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:41:09.867 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:41:09.867 139328 ERROR oslo_service.periodic_task >2018-06-25 06:41:39.749 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:41:39.749 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:41:39.756 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:41:39.756 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:41:39.791 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:41:39.791 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:41:39.791 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:41:39.791 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:41:39.791 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:41:39.791 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:41:39.791 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:41:39.793 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:41:39.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:41:39.794 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:41:49.795 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:41:49.832 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:41:49.832 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:41:49.832 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:41:49.832 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:41:49.832 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:41:49.832 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:41:49.832 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:41:49.834 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:41:49.834 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:41:49.834 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:42:09.835 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:42:09.867 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:42:09.867 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:42:09.867 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:42:09.867 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:42:09.867 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:42:09.867 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:42:09.867 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:42:09.869 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:42:09.869 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.113 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:42:09.870 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:42:09.870 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:42:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:42:09.870 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:42:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:42:09.870 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:42:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:42:09.870 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:42:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:42:09.870 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:42:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:42:09.870 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:42:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:42:09.870 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:42:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:42:09.870 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:42:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:42:09.870 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:42:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:42:09.870 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:42:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:42:09.870 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:42:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:42:09.870 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:42:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:42:09.870 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:42:09.870 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:42:09.870 139328 ERROR oslo_service.periodic_task >2018-06-25 06:42:39.751 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:42:39.752 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:42:39.759 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:42:39.760 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:42:39.793 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:42:39.793 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:42:39.793 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:42:39.793 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:42:39.793 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:42:39.793 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:42:39.793 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:42:39.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:42:39.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:42:39.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:42:49.796 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:42:49.830 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:42:49.830 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:42:49.830 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:42:49.830 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:42:49.830 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:42:49.830 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:42:49.830 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:42:49.832 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:42:49.832 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:42:49.832 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:43:09.833 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:43:09.866 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:43:09.866 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:43:09.866 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:43:09.866 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:43:09.866 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:43:09.866 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:43:09.866 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:43:09.869 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:43:09.869 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:43:09.869 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:43:09.869 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:43:09.869 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:43:09.869 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:43:09.869 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:43:09.869 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:43:09.869 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:43:09.869 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:43:09.869 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:43:09.869 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:43:09.869 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:43:09.869 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:43:09.869 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:43:09.869 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:43:09.869 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:43:09.869 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:43:09.869 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:43:09.869 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:43:09.869 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:43:09.869 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:43:09.869 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:43:09.869 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:43:09.869 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:43:09.869 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:43:09.869 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:43:09.869 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:43:09.869 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:43:09.869 139328 ERROR oslo_service.periodic_task >2018-06-25 06:43:39.751 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:43:39.751 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:43:39.757 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:43:39.758 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:43:39.794 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:43:39.794 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:43:39.794 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:43:39.794 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:43:39.794 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:43:39.794 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:43:39.794 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:43:39.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:43:39.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:43:39.796 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:43:49.797 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:43:49.830 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:43:49.830 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:43:49.830 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:43:49.830 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:43:49.830 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:43:49.830 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:43:49.830 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:43:49.833 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:43:49.833 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:43:49.833 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:44:09.834 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:44:09.867 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:44:09.867 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:44:09.867 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:44:09.867 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:44:09.867 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:44:09.867 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:44:09.867 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:44:09.869 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:44:09.870 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.112 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:44:09.870 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:44:09.870 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:44:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:44:09.870 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:44:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:44:09.870 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:44:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:44:09.870 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:44:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:44:09.870 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:44:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:44:09.870 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:44:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:44:09.870 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:44:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:44:09.870 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:44:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:44:09.870 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:44:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:44:09.870 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:44:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:44:09.870 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:44:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:44:09.870 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:44:09.870 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:44:09.870 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:44:09.870 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:44:09.870 139328 ERROR oslo_service.periodic_task >2018-06-25 06:44:39.751 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:44:39.752 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:44:39.758 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:44:39.759 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:44:39.793 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:44:39.793 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:44:39.793 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:44:39.793 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:44:39.793 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:44:39.793 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:44:39.793 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:44:39.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:44:39.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:44:39.795 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:44:49.796 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:44:49.829 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:44:49.829 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:44:49.829 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:44:49.829 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:44:49.829 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:44:49.829 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:44:49.829 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:44:49.832 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:44:49.832 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:44:49.833 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:45:09.833 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:45:09.871 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:45:09.871 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:45:09.871 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:45:09.871 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:45:09.871 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:45:09.871 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:45:09.871 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:45:09.873 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:45:09.874 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:45:09.874 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:45:09.874 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:45:09.874 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:45:09.874 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:45:09.874 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:45:09.874 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:45:09.874 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:45:09.874 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:45:09.874 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:45:09.874 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:45:09.874 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:45:09.874 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:45:09.874 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:45:09.874 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:45:09.874 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:45:09.874 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:45:09.874 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:45:09.874 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:45:09.874 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:45:09.874 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:45:09.874 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:45:09.874 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:45:09.874 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:45:09.874 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:45:09.874 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:45:09.874 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:45:09.874 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:45:09.874 139328 ERROR oslo_service.periodic_task >2018-06-25 06:45:39.756 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:45:39.756 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:45:39.770 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:45:39.771 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:45:39.805 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:45:39.805 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:45:39.805 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:45:39.805 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:45:39.805 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:45:39.805 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:45:39.805 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:45:39.807 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:45:39.808 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:45:39.808 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:45:49.809 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:45:49.845 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:45:49.845 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:45:49.845 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:45:49.845 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:45:49.845 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:45:49.845 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:45:49.845 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:45:49.849 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:45:49.849 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:45:49.850 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:46:09.850 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:46:09.884 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:46:09.884 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:46:09.884 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:46:09.884 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:46:09.884 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:46:09.884 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:46:09.884 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:46:09.886 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:46:09.886 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.115 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:46:09.886 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:46:09.886 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:46:09.886 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:46:09.886 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:46:09.886 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:46:09.886 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:46:09.886 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:46:09.886 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:46:09.886 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:46:09.886 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:46:09.886 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:46:09.886 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:46:09.886 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:46:09.886 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:46:09.886 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:46:09.886 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:46:09.886 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:46:09.886 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:46:09.886 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:46:09.886 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:46:09.886 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:46:09.886 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:46:09.886 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:46:09.886 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:46:09.886 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:46:09.886 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:46:09.886 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:46:09.886 139328 ERROR oslo_service.periodic_task >2018-06-25 06:46:39.758 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:46:39.759 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:46:39.795 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:46:39.795 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:46:39.795 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:46:39.795 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:46:39.795 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:46:39.795 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:46:39.795 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:46:39.797 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:46:39.797 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:46:39.798 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:46:49.798 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:46:49.849 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:46:49.849 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:46:49.849 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:46:49.849 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:46:49.849 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:46:49.849 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:46:49.849 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:46:49.852 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:46:49.852 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.093 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:46:49.853 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:47:09.853 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:47:09.886 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:47:09.886 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:47:09.886 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:47:09.886 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:47:09.886 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:47:09.886 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:47:09.886 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:47:09.888 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:47:09.888 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.129 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:47:09.888 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:47:09.888 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:47:09.888 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:47:09.888 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:47:09.888 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:47:09.888 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:47:09.888 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:47:09.888 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:47:09.888 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:47:09.888 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:47:09.888 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:47:09.888 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:47:09.888 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:47:09.888 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:47:09.888 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:47:09.888 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:47:09.888 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:47:09.888 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:47:09.888 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:47:09.888 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:47:09.888 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:47:09.888 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:47:09.888 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:47:09.888 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:47:09.888 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:47:09.888 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:47:09.888 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:47:09.888 139328 ERROR oslo_service.periodic_task >2018-06-25 06:47:39.760 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:47:39.760 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:47:39.769 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:47:39.770 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:47:39.803 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:47:39.803 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:47:39.803 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:47:39.803 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:47:39.803 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:47:39.803 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:47:39.803 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:47:39.805 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:47:39.805 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:47:39.805 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:47:49.806 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:47:49.843 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:47:49.843 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:47:49.843 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:47:49.843 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:47:49.843 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:47:49.843 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:47:49.843 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:47:49.845 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:47:49.845 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:47:49.846 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:48:09.846 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:48:09.879 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:48:09.879 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:48:09.879 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:48:09.879 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:48:09.879 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:48:09.879 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:48:09.879 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:48:09.881 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:48:09.881 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:48:09.881 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:48:09.881 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:48:09.881 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:48:09.881 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:48:09.881 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:48:09.881 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:48:09.881 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:48:09.881 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:48:09.881 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:48:09.881 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:48:09.881 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:48:09.881 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:48:09.881 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:48:09.881 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:48:09.881 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:48:09.881 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:48:09.881 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:48:09.881 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:48:09.881 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:48:09.881 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:48:09.881 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:48:09.881 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:48:09.881 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:48:09.881 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:48:09.881 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:48:09.881 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:48:09.881 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:48:09.881 139328 ERROR oslo_service.periodic_task >2018-06-25 06:48:39.763 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:48:39.764 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:48:39.771 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:48:39.771 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:48:39.804 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:48:39.804 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:48:39.804 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:48:39.804 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:48:39.804 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:48:39.804 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:48:39.804 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:48:39.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:48:39.806 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:48:39.807 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:48:49.807 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:48:49.842 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:48:49.842 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:48:49.842 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:48:49.842 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:48:49.842 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:48:49.842 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:48:49.842 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:48:49.846 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:48:49.847 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:48:49.847 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:49:09.847 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:49:09.880 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:49:09.880 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:49:09.880 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:49:09.880 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:49:09.880 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:49:09.880 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:49:09.880 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:49:09.882 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:49:09.882 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:49:09.882 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:49:09.882 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:49:09.882 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:49:09.882 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:49:09.882 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:49:09.882 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:49:09.882 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:49:09.882 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:49:09.882 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:49:09.882 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:49:09.882 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:49:09.882 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:49:09.882 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:49:09.882 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:49:09.882 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:49:09.882 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:49:09.882 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:49:09.882 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:49:09.882 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:49:09.882 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:49:09.882 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:49:09.882 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:49:09.882 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:49:09.882 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:49:09.882 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:49:09.882 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:49:09.882 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:49:09.882 139328 ERROR oslo_service.periodic_task >2018-06-25 06:49:39.764 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:49:39.764 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:49:39.771 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:49:39.772 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:49:39.804 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:49:39.804 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:49:39.804 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:49:39.804 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:49:39.804 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:49:39.804 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:49:39.804 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:49:39.807 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:49:39.807 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:49:39.808 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:49:49.808 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:49:49.841 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:49:49.841 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:49:49.841 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:49:49.841 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:49:49.841 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:49:49.841 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:49:49.841 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:49:49.843 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:49:49.843 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.072 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:49:49.844 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:50:09.844 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:50:09.896 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:50:09.896 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:50:09.896 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:50:09.896 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:50:09.896 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:50:09.896 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:50:09.896 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:50:09.899 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:50:09.899 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.127 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:50:09.900 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:50:09.900 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:50:09.900 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:50:09.900 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:50:09.900 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:50:09.900 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:50:09.900 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:50:09.900 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:50:09.900 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:50:09.900 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:50:09.900 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:50:09.900 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:50:09.900 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:50:09.900 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:50:09.900 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:50:09.900 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:50:09.900 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:50:09.900 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:50:09.900 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:50:09.900 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:50:09.900 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:50:09.900 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:50:09.900 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:50:09.900 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:50:09.900 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:50:09.900 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:50:09.900 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:50:09.900 139328 ERROR oslo_service.periodic_task >2018-06-25 06:50:39.761 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:50:39.762 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:50:39.771 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:50:39.772 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:50:39.807 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:50:39.807 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:50:39.807 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:50:39.807 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:50:39.807 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:50:39.807 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:50:39.807 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:50:39.809 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:50:39.809 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:50:39.810 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:50:49.810 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:50:49.844 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:50:49.844 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:50:49.844 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:50:49.844 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:50:49.844 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:50:49.844 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:50:49.844 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:50:49.847 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:50:49.847 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:50:49.848 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:51:09.848 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:51:09.880 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:51:09.880 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:51:09.880 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:51:09.880 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:51:09.880 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:51:09.880 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:51:09.880 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:51:09.882 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:51:09.882 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.111 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:51:09.883 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:51:09.883 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:51:09.883 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:51:09.883 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:51:09.883 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:51:09.883 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:51:09.883 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:51:09.883 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:51:09.883 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:51:09.883 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:51:09.883 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:51:09.883 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:51:09.883 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:51:09.883 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:51:09.883 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:51:09.883 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:51:09.883 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:51:09.883 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:51:09.883 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:51:09.883 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:51:09.883 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:51:09.883 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:51:09.883 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:51:09.883 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:51:09.883 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:51:09.883 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:51:09.883 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:51:09.883 139328 ERROR oslo_service.periodic_task >2018-06-25 06:51:39.764 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:51:39.765 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:51:39.773 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:51:39.773 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:51:39.817 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:51:39.817 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:51:39.817 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:51:39.817 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:51:39.817 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:51:39.817 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:51:39.817 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:51:39.820 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:51:39.820 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.046 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:51:39.821 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:51:49.822 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:51:49.863 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:51:49.863 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:51:49.863 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:51:49.863 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:51:49.863 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:51:49.863 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:51:49.863 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:51:49.865 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:51:49.865 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.091 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:51:49.865 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:52:09.866 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:52:09.902 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:52:09.902 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:52:09.902 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:52:09.902 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:52:09.902 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:52:09.902 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:52:09.902 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:52:09.904 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:52:09.905 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.131 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:52:09.905 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:52:09.905 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:52:09.905 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:52:09.905 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:52:09.905 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:52:09.905 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:52:09.905 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:52:09.905 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:52:09.905 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:52:09.905 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:52:09.905 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:52:09.905 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:52:09.905 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:52:09.905 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:52:09.905 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:52:09.905 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:52:09.905 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:52:09.905 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:52:09.905 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:52:09.905 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:52:09.905 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:52:09.905 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:52:09.905 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:52:09.905 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:52:09.905 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:52:09.905 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:52:09.905 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:52:09.905 139328 ERROR oslo_service.periodic_task >2018-06-25 06:52:39.766 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:52:39.768 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:52:39.776 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:52:39.777 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:52:39.812 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:52:39.812 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:52:39.812 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:52:39.812 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:52:39.812 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:52:39.812 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:52:39.812 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:52:39.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:52:39.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:52:39.815 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:52:49.816 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:52:49.849 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:52:49.849 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:52:49.849 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:52:49.849 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:52:49.849 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:52:49.849 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:52:49.849 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:52:49.851 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:52:49.851 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:52:49.851 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:53:09.852 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:53:09.884 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:53:09.884 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:53:09.884 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:53:09.884 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:53:09.884 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:53:09.884 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:53:09.884 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:53:09.886 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:53:09.886 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.11 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:53:09.887 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:53:09.887 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:53:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:53:09.887 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:53:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:53:09.887 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:53:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:53:09.887 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:53:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:53:09.887 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:53:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:53:09.887 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:53:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:53:09.887 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:53:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:53:09.887 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:53:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:53:09.887 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:53:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:53:09.887 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:53:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:53:09.887 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:53:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:53:09.887 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:53:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:53:09.887 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:53:09.887 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:53:09.887 139328 ERROR oslo_service.periodic_task >2018-06-25 06:53:39.768 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:53:39.769 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:53:39.776 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:53:39.777 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:53:39.814 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:53:39.814 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:53:39.814 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:53:39.814 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:53:39.814 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:53:39.814 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:53:39.814 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:53:39.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:53:39.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.039 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:53:39.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:53:49.818 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:53:49.853 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:53:49.853 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:53:49.853 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:53:49.853 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:53:49.853 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:53:49.853 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:53:49.853 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:53:49.855 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:53:49.856 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.078 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:53:49.857 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:54:09.858 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:54:09.894 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:54:09.894 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:54:09.894 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:54:09.894 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:54:09.894 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:54:09.894 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:54:09.894 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:54:09.896 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:54:09.897 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.119 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:54:09.897 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:54:09.897 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:54:09.897 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:54:09.897 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:54:09.897 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:54:09.897 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:54:09.897 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:54:09.897 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:54:09.897 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:54:09.897 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:54:09.897 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:54:09.897 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:54:09.897 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:54:09.897 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:54:09.897 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:54:09.897 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:54:09.897 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:54:09.897 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:54:09.897 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:54:09.897 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:54:09.897 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:54:09.897 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:54:09.897 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:54:09.897 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:54:09.897 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:54:09.897 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:54:09.897 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:54:09.897 139328 ERROR oslo_service.periodic_task >2018-06-25 06:54:39.768 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:54:39.769 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:54:39.777 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:54:39.778 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:54:39.811 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:54:39.811 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:54:39.811 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:54:39.811 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:54:39.811 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:54:39.811 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:54:39.811 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:54:39.813 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:54:39.814 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:54:39.814 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:54:49.814 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:54:49.848 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:54:49.848 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:54:49.848 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:54:49.848 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:54:49.848 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:54:49.848 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:54:49.848 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:54:49.850 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:54:49.851 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:54:49.851 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:55:09.852 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:55:09.884 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:55:09.884 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:55:09.884 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:55:09.884 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:55:09.884 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:55:09.884 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:55:09.884 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:55:09.886 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:55:09.886 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:55:09.887 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:55:09.887 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:55:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:55:09.887 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:55:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:55:09.887 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:55:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:55:09.887 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:55:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:55:09.887 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:55:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:55:09.887 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:55:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:55:09.887 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:55:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:55:09.887 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:55:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:55:09.887 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:55:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:55:09.887 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:55:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:55:09.887 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:55:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:55:09.887 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:55:09.887 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:55:09.887 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:55:09.887 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:55:09.887 139328 ERROR oslo_service.periodic_task >2018-06-25 06:55:39.768 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:55:39.769 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:55:39.776 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:55:39.777 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:55:39.809 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:55:39.809 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:55:39.809 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:55:39.809 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:55:39.809 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:55:39.809 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:55:39.809 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:55:39.811 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:55:39.811 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.034 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:55:39.812 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:55:49.812 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:55:49.854 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:55:49.854 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:55:49.854 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:55:49.854 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:55:49.854 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:55:49.854 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:55:49.854 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:55:49.856 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:55:49.856 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:55:49.857 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:56:09.857 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:56:09.893 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:56:09.893 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:56:09.893 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:56:09.893 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:56:09.893 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:56:09.893 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:56:09.893 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:56:09.895 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:56:09.896 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.119 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:56:09.896 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:56:09.896 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:56:09.896 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:56:09.896 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:56:09.896 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:56:09.896 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:56:09.896 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:56:09.896 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:56:09.896 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:56:09.896 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:56:09.896 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:56:09.896 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:56:09.896 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:56:09.896 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:56:09.896 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:56:09.896 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:56:09.896 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:56:09.896 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:56:09.896 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:56:09.896 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:56:09.896 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:56:09.896 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:56:09.896 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:56:09.896 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:56:09.896 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:56:09.896 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:56:09.896 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:56:09.896 139328 ERROR oslo_service.periodic_task >2018-06-25 06:56:39.768 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:56:39.769 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:56:39.776 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:56:39.776 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:56:39.811 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:56:39.811 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:56:39.811 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:56:39.811 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:56:39.811 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:56:39.811 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:56:39.811 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:56:39.814 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:56:39.814 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.037 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:56:39.814 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:56:49.815 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:56:49.852 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:56:49.852 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:56:49.852 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:56:49.852 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:56:49.852 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:56:49.852 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:56:49.852 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:56:49.854 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:56:49.854 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.077 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:56:49.855 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:57:09.855 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:57:09.889 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:57:09.889 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:57:09.889 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:57:09.889 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:57:09.889 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:57:09.889 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:57:09.889 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:57:09.890 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:57:09.891 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:57:09.891 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:57:09.891 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:57:09.891 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:57:09.891 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:57:09.891 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:57:09.891 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:57:09.891 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:57:09.891 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:57:09.891 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:57:09.891 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:57:09.891 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:57:09.891 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:57:09.891 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:57:09.891 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:57:09.891 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:57:09.891 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:57:09.891 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:57:09.891 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:57:09.891 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:57:09.891 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:57:09.891 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:57:09.891 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:57:09.891 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:57:09.891 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:57:09.891 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:57:09.891 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:57:09.891 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:57:09.891 139328 ERROR oslo_service.periodic_task >2018-06-25 06:57:39.772 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:57:39.773 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:57:39.781 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:57:39.781 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:57:39.815 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:57:39.815 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:57:39.815 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:57:39.815 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:57:39.815 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:57:39.815 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:57:39.815 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:57:39.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:57:39.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:57:39.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:57:49.818 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:57:49.853 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:57:49.853 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:57:49.853 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:57:49.853 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:57:49.853 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:57:49.853 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:57:49.853 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:57:49.855 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:57:49.856 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.074 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:57:49.856 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:58:09.857 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:58:09.900 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:58:09.900 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:58:09.900 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:58:09.900 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:58:09.900 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:58:09.900 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:58:09.900 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:58:09.904 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:58:09.904 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.122 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:58:09.905 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:58:09.905 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:58:09.905 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:58:09.905 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:58:09.905 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:58:09.905 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:58:09.905 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:58:09.905 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:58:09.905 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:58:09.905 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:58:09.905 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:58:09.905 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:58:09.905 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:58:09.905 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:58:09.905 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:58:09.905 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:58:09.905 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:58:09.905 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:58:09.905 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:58:09.905 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:58:09.905 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:58:09.905 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:58:09.905 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:58:09.905 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:58:09.905 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:58:09.905 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:58:09.905 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:58:09.905 139328 ERROR oslo_service.periodic_task >2018-06-25 06:58:39.776 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:58:39.777 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:58:39.784 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:58:39.784 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:58:39.819 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:58:39.819 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:58:39.819 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:58:39.819 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:58:39.819 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:58:39.819 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:58:39.819 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:58:39.821 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:58:39.821 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:58:39.822 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:58:49.822 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:58:49.856 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:58:49.856 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:58:49.856 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:58:49.856 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:58:49.856 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:58:49.856 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:58:49.856 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:58:49.858 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:58:49.858 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:58:49.858 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:59:09.859 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:59:09.892 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:59:09.892 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:59:09.892 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:59:09.892 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:59:09.892 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:59:09.892 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:59:09.892 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:59:09.894 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:59:09.894 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.109 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:59:09.895 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:59:09.895 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 06:59:09.895 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 06:59:09.895 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 06:59:09.895 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 06:59:09.895 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 06:59:09.895 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 06:59:09.895 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 06:59:09.895 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 06:59:09.895 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 06:59:09.895 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 06:59:09.895 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 06:59:09.895 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 06:59:09.895 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 06:59:09.895 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 06:59:09.895 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 06:59:09.895 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 06:59:09.895 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 06:59:09.895 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 06:59:09.895 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 06:59:09.895 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 06:59:09.895 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 06:59:09.895 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 06:59:09.895 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 06:59:09.895 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 06:59:09.895 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 06:59:09.895 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 06:59:09.895 139328 ERROR oslo_service.periodic_task >2018-06-25 06:59:39.776 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:59:39.777 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 06:59:39.784 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 06:59:39.784 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:59:39.820 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:59:39.820 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:59:39.820 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:59:39.820 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:59:39.820 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:59:39.820 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:59:39.820 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:59:39.822 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:59:39.822 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.038 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:59:39.823 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 06:59:49.823 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 06:59:49.857 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 06:59:49.857 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 06:59:49.857 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 06:59:49.857 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 06:59:49.857 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 06:59:49.857 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 06:59:49.857 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 06:59:49.859 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 06:59:49.860 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.075 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 06:59:49.860 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 07:00:09.860 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 07:00:09.900 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 07:00:09.900 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 07:00:09.900 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 07:00:09.900 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 07:00:09.900 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 07:00:09.900 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 07:00:09.900 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 07:00:09.902 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 07:00:09.902 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.118 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 07:00:09.903 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 07:00:09.903 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 07:00:09.903 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 07:00:09.903 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 07:00:09.903 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 07:00:09.903 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 07:00:09.903 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 07:00:09.903 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 07:00:09.903 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 07:00:09.903 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 07:00:09.903 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 07:00:09.903 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 07:00:09.903 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 07:00:09.903 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 07:00:09.903 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 07:00:09.903 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 07:00:09.903 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 07:00:09.903 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 07:00:09.903 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 07:00:09.903 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 07:00:09.903 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 07:00:09.903 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 07:00:09.903 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 07:00:09.903 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 07:00:09.903 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 07:00:09.903 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 07:00:09.903 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 07:00:09.903 139328 ERROR oslo_service.periodic_task >2018-06-25 07:00:39.774 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 07:00:39.775 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 07:00:39.782 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 07:00:39.783 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 07:00:39.815 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 07:00:39.815 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 07:00:39.815 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 07:00:39.815 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 07:00:39.815 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 07:00:39.815 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 07:00:39.815 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 07:00:39.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 07:00:39.817 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.035 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 07:00:39.818 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 07:00:49.818 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 07:00:49.853 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 07:00:49.853 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 07:00:49.853 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 07:00:49.853 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 07:00:49.853 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 07:00:49.853 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 07:00:49.853 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 07:00:49.855 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 07:00:49.856 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.073 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 07:00:49.856 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 07:01:09.856 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 07:01:09.894 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 07:01:09.894 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 07:01:09.894 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 07:01:09.894 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 07:01:09.894 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 07:01:09.894 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 07:01:09.894 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 07:01:09.896 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 07:01:09.897 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.114 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 07:01:09.897 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 07:01:09.897 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 07:01:09.897 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 07:01:09.897 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 07:01:09.897 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 07:01:09.897 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 07:01:09.897 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 07:01:09.897 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 07:01:09.897 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 07:01:09.897 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 07:01:09.897 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 07:01:09.897 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 07:01:09.897 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 07:01:09.897 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 07:01:09.897 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 07:01:09.897 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 07:01:09.897 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 07:01:09.897 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 07:01:09.897 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 07:01:09.897 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 07:01:09.897 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 07:01:09.897 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 07:01:09.897 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 07:01:09.897 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 07:01:09.897 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 07:01:09.897 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 07:01:09.897 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 07:01:09.897 139328 ERROR oslo_service.periodic_task >2018-06-25 07:01:39.779 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 07:01:39.780 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 07:01:39.788 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 07:01:39.788 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 07:01:39.827 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 07:01:39.827 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 07:01:39.827 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 07:01:39.827 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 07:01:39.827 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 07:01:39.827 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 07:01:39.827 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 07:01:39.830 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 07:01:39.830 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.041 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 07:01:39.830 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 07:01:49.831 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 07:01:49.866 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 07:01:49.866 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 07:01:49.866 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 07:01:49.866 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 07:01:49.866 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 07:01:49.866 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 07:01:49.866 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 07:01:49.868 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 07:01:49.868 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.079 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 07:01:49.868 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 07:02:09.869 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 07:02:09.907 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 07:02:09.907 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 07:02:09.907 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 07:02:09.907 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 07:02:09.907 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 07:02:09.907 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 07:02:09.907 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 07:02:09.910 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 07:02:09.910 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.122 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 07:02:09.911 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 07:02:09.911 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >2018-06-25 07:02:09.911 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks >2018-06-25 07:02:09.911 139328 ERROR oslo_service.periodic_task task(self, context) >2018-06-25 07:02:09.911 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2367, in _report_driver_status >2018-06-25 07:02:09.911 139328 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True) >2018-06-25 07:02:09.911 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats >2018-06-25 07:02:09.911 139328 ERROR oslo_service.periodic_task self._update_volume_stats() >2018-06-25 07:02:09.911 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats >2018-06-25 07:02:09.911 139328 ERROR oslo_service.periodic_task free_capacity, total_capacity = self._get_pool_stats() >2018-06-25 07:02:09.911 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 403, in _get_pool_stats >2018-06-25 07:02:09.911 139328 ERROR oslo_service.periodic_task with RADOSClient(self) as client: >2018-06-25 07:02:09.911 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 154, in __init__ >2018-06-25 07:02:09.911 139328 ERROR oslo_service.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool) >2018-06-25 07:02:09.911 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 330, in _connect_to_rados >2018-06-25 07:02:09.911 139328 ERROR oslo_service.periodic_task return _do_conn(pool, remote, timeout) >2018-06-25 07:02:09.911 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper >2018-06-25 07:02:09.911 139328 ERROR oslo_service.periodic_task return r.call(f, *args, **kwargs) >2018-06-25 07:02:09.911 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call >2018-06-25 07:02:09.911 139328 ERROR oslo_service.periodic_task raise attempt.get() >2018-06-25 07:02:09.911 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get >2018-06-25 07:02:09.911 139328 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) >2018-06-25 07:02:09.911 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call >2018-06-25 07:02:09.911 139328 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) >2018-06-25 07:02:09.911 139328 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 328, in _do_conn >2018-06-25 07:02:09.911 139328 ERROR oslo_service.periodic_task raise exception.VolumeBackendAPIException(data=msg) >2018-06-25 07:02:09.911 139328 ERROR oslo_service.periodic_task VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 07:02:09.911 139328 ERROR oslo_service.periodic_task >2018-06-25 07:02:39.782 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 07:02:39.783 139328 DEBUG cinder.manager [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:189 >2018-06-25 07:02:39.796 139328 DEBUG oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 >2018-06-25 07:02:39.797 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 07:02:39.830 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 07:02:39.830 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 07:02:39.830 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 07:02:39.830 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 07:02:39.830 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 07:02:39.830 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 07:02:39.830 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 07:02:39.833 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 07:02:39.833 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 0.036 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 07:02:39.834 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 10.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 07:02:49.834 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 07:02:49.876 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 07:02:49.876 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 07:02:49.876 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 07:02:49.876 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 07:02:49.876 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 07:02:49.876 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 07:02:49.876 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 07:02:49.878 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 2 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 07:02:49.878 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 10.081 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 07:02:49.878 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Sleeping for 20.0 seconds _backoff_sleep /usr/lib/python2.7/site-packages/cinder/utils.py:777 >2018-06-25 07:03:09.879 139328 DEBUG cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308 >2018-06-25 07:03:09.914 139328 ERROR cinder.volume.drivers.rbd [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error connecting to ceph cluster.: Error: error connecting to the cluster: error code 95 >2018-06-25 07:03:09.914 139328 ERROR cinder.volume.drivers.rbd Traceback (most recent call last): >2018-06-25 07:03:09.914 139328 ERROR cinder.volume.drivers.rbd File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 321, in _do_conn >2018-06-25 07:03:09.914 139328 ERROR cinder.volume.drivers.rbd client.connect() >2018-06-25 07:03:09.914 139328 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) >2018-06-25 07:03:09.914 139328 ERROR cinder.volume.drivers.rbd Error: error connecting to the cluster: error code 95 >2018-06-25 07:03:09.914 139328 ERROR cinder.volume.drivers.rbd >2018-06-25 07:03:09.916 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Failed attempt 3 _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:783 >2018-06-25 07:03:09.917 139328 DEBUG cinder.utils [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Have been at this for 30.12 seconds _print_stop /usr/lib/python2.7/site-packages/cinder/utils.py:785 >2018-06-25 07:03:09.917 139328 ERROR oslo_service.periodic_task [req-e679111c-4839-434c-8828-1b058c0edb30 - - - - -] Error during VolumeManager._report_driver_status: VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. >2018-06-25 07:03:09.917 139328 ERROR oslo_service.periodic_task Traceback (most recent call last): >